Josh Davies

Senior Software Engineer

I build beautiful, engaging, and perfectly executed frontend and backend experiences. While finding time to write, hike and read.

Blog

The Job Ran Before the Row Existed

  • laravel
  • typescript
  • databases
  • queues

Here's a bug report a lot of apps get eventually: the welcome email job failed with ModelNotFoundException. But the user is right there in the database.

Both things are true. The user exists now. They didn't exist yet when the job looked.

The transaction isn't done until it commits

DB::transaction(function () use ($data) {
    $user = User::create($data);
    $user->profile()->create(['timezone' => $data['timezone']]);
 
    SendWelcomeEmail::dispatch($user);
});

Reads fine. Create the user, create the profile, queue the email.

The problem is that dispatch() pushes the job to Redis (or SQS, or whatever you use) right now, while the transaction is still open. Redis doesn't know or care about your MySQL transaction.

An idle worker grabs the job in a few milliseconds and tries to load the user. The database says there's no such row, because to every other connection that row doesn't exist until COMMIT.

On your laptop the timing usually works out, so it passes every time. In production, with workers sitting on a fast queue, it fails some of the time. Which is the worst amount.

Sometimes the row never exists

The other direction is worse. If something after the dispatch throws, the transaction rolls back. The user and the profile are gone. The job is not. It's already in the queue.

And if that side effect wasn't a job but a direct call (sending the email right there, charging a card, posting to a CRM), it already happened. You can roll back a row. You can't roll back an email.

Laravel will wait if you ask

SendWelcomeEmail::dispatch($user)->afterCommit();

Now the job is held until the outermost transaction commits. If it rolls back, the job is thrown away.

If you want that everywhere, turn it on for the queue connection in config/queue.php:

'redis' => [
    'driver' => 'redis',
    // ...
    'after_commit' => true,
],

With that set, jobs dispatched inside a transaction wait by default, and you can opt out with ->beforeCommit() when you actually mean it. Queued listeners, mail, notifications, and broadcast events follow the same setting.

For things that aren't queued, there's DB::afterCommit():

DB::transaction(function () use ($order) {
    $order->update(['status' => 'paid']);
 
    DB::afterCommit(fn () => Cache::forget("order:{$order->id}"));
});

That one matters more than it looks. Clear the cache before the commit and another request can reload the old value into it before your change lands. Now the cache is stale and nobody is going to clear it again.

Model events are the sneaky version of this, because save() fires them from inside whatever transaction you're in. Queued listeners can implement ShouldQueueAfterCommit, and observers can implement ShouldHandleEventsAfterCommit:

class SyncOrderToCrm implements ShouldQueueAfterCommit
{
    public function handle(OrderPlaced $event): void
    {
        // runs only once the order is actually committed
    }
}

Keep the slow stuff out of the transaction

Waiting for the commit fixes the timing. It doesn't fix this:

DB::transaction(function () use ($orderId, $stripe) {
    $order = Order::lockForUpdate()->findOrFail($orderId);
 
    $intent = $stripe->paymentIntents->create([
        'amount' => $order->total_cents,
        'currency' => 'usd',
        'customer' => $order->stripe_customer_id,
    ]);
 
    $order->update(['status' => 'paid', 'payment_intent_id' => $intent->id]);
});

Two problems. The transaction holds a row lock and a database connection the whole time it waits on the network, so every other request that touches that order waits too. And if the final update() fails after Stripe said yes, the rollback undoes your record of the charge. Not the charge.

A better shape is: claim it, commit, call out, record the result.

$claimed = Order::whereKey($orderId)
    ->where('status', 'pending')
    ->update(['status' => 'charging']);
 
if (! $claimed) {
    return; // someone else already has it
}
 
$order = Order::findOrFail($orderId);
 
$intent = $stripe->paymentIntents->create([
    'amount' => $order->total_cents,
    'currency' => 'usd',
    'customer' => $order->stripe_customer_id,
], ['idempotency_key' => "order-{$orderId}"]);
 
$order->update(['status' => 'paid', 'payment_intent_id' => $intent->id]);

Every write is short and commits on its own. If the process dies in the middle, you have an order stuck in charging that you can find and reconcile. The idempotency key means a retry doesn't charge twice.

Node has the same trap

Prisma's interactive transactions look just like the Laravel version:

await prisma.$transaction(async (tx) => {
  const user = await tx.user.create({ data })
  await tx.profile.create({ data: { userId: user.id } })
 
  await sendWelcomeEmail(user.email) // runs before commit
})

Same two problems, plus a fun one. Interactive transactions have a timeout (5 seconds by default). A slow email provider can push you past it, Prisma rolls the transaction back, and the email has already gone out. For a user who doesn't exist.

Return what you need from the transaction and do the side effects after:

const user = await prisma.$transaction(async (tx) => {
  const user = await tx.user.create({ data })
  await tx.profile.create({ data: { userId: user.id } })
  return user
})
 
await welcomeQueue.add('welcome-email', { userId: user.id }) // BullMQ

When "after commit" isn't enough

There's still a gap. The commit succeeds, then the process dies (a deploy, an out of memory kill, Redis blinks) before the job gets pushed. The row is there. The job never is.

For a welcome email, that's probably fine. For "tell the warehouse to ship this," maybe not.

That's what an outbox table is for. You write the message in the same transaction as the data, and a separate worker sends it:

await prisma.$transaction(async (tx) => {
  const order = await tx.order.create({ data: orderData })
 
  await tx.outboxMessage.create({
    data: { type: 'order.placed', payload: { orderId: order.id } },
  })
})

outboxMessage is a model you define, not something Prisma ships. The point is that the order and the message commit together or not at all. A worker picks up rows where sentAt is null, sends them, and marks them sent.

That worker can still send something twice if it dies between sending and marking. So the receiving side needs to be safe to run twice. You don't get to skip that part. You just get to choose where it lives.

The habit

Any time you see a dispatch, a Mail, an Http call, a webhook, or a cache clear inside a transaction closure, ask two questions. What happens if this runs before the commit? What happens if the commit never comes?

Database writes go inside the transaction. Telling the rest of the world about them goes after.

← All posts