Josh Davies

Senior Software Engineer

I build beautiful, engaging, and perfectly executed frontend and backend experiences. While finding time to write, hike and read.

Blog

Your Worker Remembers the Last Job

  • laravel
  • php
  • typescript
  • node
  • queues

Classic PHP spoiled us. Under PHP-FPM, every request boots the app, does its thing, and throws all of your app's state away. Set a static property, change a config value, stash something on a singleton. Gone by the next request.

Then you run php artisan queue:work.

That worker boots Laravel once and then processes job after job in the same process. Hundreds. Thousands. Octane does the same thing for HTTP requests. And Node has always worked this way.

So anything one job leaves lying around, the next job picks up. Nobody planned that. It just happens.

The locale that wouldn't leave

Here's a job that looks completely reasonable:

class SendWelcomeEmail implements ShouldQueue
{
    use Queueable;
 
    public function __construct(public User $user) {}
 
    public function handle(): void
    {
        App::setLocale($this->user->locale);
 
        Mail::to($this->user)->send(new WelcomeMail($this->user));
    }
}

The French user gets a French email. Great.

But App::setLocale() changes the locale for the whole application, and the application is the worker process. The next job in that worker runs in French too. So does the one after it. Your English password reset now goes out in French because a welcome email ran first.

You'll never see this locally with one job at a time. You'll see it in production as "some emails come out in the wrong language, sometimes."

Laravel already has a better way. Tell the mail what locale to use instead of changing the global one:

Mail::to($this->user)
    ->locale($this->user->locale)
    ->send(new WelcomeMail($this->user));

Or implement HasLocalePreference on your User model, and mailables and notifications pick up the user's locale on their own.

For global state with no built in escape hatch, put it back when you're done:

$previous = App::getLocale();
App::setLocale($this->user->locale);
 
try {
    $this->buildInvoicePdf();
} finally {
    App::setLocale($previous);
}

The finally matters. If the job throws, the worker keeps going, and so does whatever you forgot to reset.

Singletons live as long as the process

A singleton in a long running worker isn't "one per request." It's one per process.

class TenantContext
{
    private ?Tenant $tenant = null;
 
    public function set(Tenant $tenant): void
    {
        $this->tenant = $tenant;
    }
 
    public function get(): Tenant
    {
        return $this->tenant ?? throw new RuntimeException('No tenant set');
    }
}
 
// AppServiceProvider
$this->app->singleton(TenantContext::class);

That RuntimeException looks like a safety check. In a worker, it mostly isn't. A job for tenant A sets the context. The next job forgets to call set(). It doesn't throw. It quietly runs against tenant A's data.

That's not a bug report. That's a data leak.

Laravel has a binding for exactly this:

$this->app->scoped(TenantContext::class);

A scoped instance gets flushed whenever Laravel starts a new lifecycle: a new job in a queue worker, a new request in Octane. Now the forgetful job hits the exception, which is what you wanted all along.

Octane adds one more trap. If you inject the request (or the container) into a singleton's constructor, that singleton keeps the request it was built with. Every later request sees the old one. Resolve the request when you need it, not once at boot.

Node does this concurrently

Node is worse, because it isn't one request after another. It's many requests at once in the same process.

let currentTenantId: string | null = null;
 
app.use((req, _res, next) => {
  currentTenantId = req.user.tenantId;
  next();
});

Request A sets the tenant and awaits a database call. While it waits, request B comes in and overwrites currentTenantId. Request A wakes up and finishes its work with B's tenant.

Works perfectly in dev, where you're the only user. Falls apart the first busy afternoon.

Module scope is shared by every request. If a value belongs to one request, it needs to travel with that request. AsyncLocalStorage does that:

import { AsyncLocalStorage } from 'node:async_hooks';
 
type RequestContext = { tenantId: string };
 
export const requestContext = new AsyncLocalStorage<RequestContext>();
 
app.use((req, _res, next) => {
  requestContext.run({ tenantId: req.user.tenantId }, next);
});
 
export function tenantId(): string {
  const ctx = requestContext.getStore();
  if (!ctx) throw new Error('No request context');
  return ctx.tenantId;
}

Each request gets its own store, and it follows that request through every await. Or skip the magic and pass the tenant as an argument. Boring, explicit, hard to get wrong.

The same goes for serverless. Lambda reuses execution environments between invocations, so module scope there survives too.

Caches that only grow

The other way long running processes bite you is slower. Memory.

const rates = new Map<string, number>(); // nothing ever leaves

In a request that lives 200 milliseconds, who cares. In a process that lives three weeks, that map is a slow leak with a nice name. The PHP version is a private static array $cache = []; that a worker keeps appending to.

If you need an in process cache, give it a ceiling:

import { LRUCache } from 'lru-cache';
 
const rates = new LRUCache<string, number>({
  max: 1000,
  ttl: 1000 * 60 * 5,
});

Or put it in Redis, where it's shared across processes and has an expiry anyway.

Restart on purpose

You will miss something. Plan for it.

php artisan queue:work --max-jobs=500 --max-time=3600 --memory=256
php artisan octane:start --max-requests=500

The worker exits after 500 jobs, an hour, or once it's past 256 MB (checked between jobs), and Supervisor starts a fresh one. Octane recycles workers after a set number of requests. PM2 has max_memory_restart for Node.

That's a seatbelt, not a fix. It limits how long leaked state hangs around. It doesn't stop the job right after the leak from getting it.

Test it like a worker

Most tests run one job and stop. Workers don't. So test what a job leaves behind:

it('does not change the global locale', function () {
    Mail::fake();
 
    $user = User::factory()->create(['locale' => 'fr']);
 
    (new SendWelcomeEmail($user))->handle();
 
    expect(App::getLocale())->toBe(config('app.locale'));
});

Cheap to write. It fails against the first version of the job and passes against the fixed one.

The question to ask

Before you set a static, register a singleton, call something with "set" in the name, or declare a let at the top of a Node module, ask one thing:

If this process runs a thousand of these, what does number 1,000 see that number 1 left behind?

If the answer is "something," scope it, reset it, or pass it in.

← All posts