Josh Davies

Senior Software Engineer

I build beautiful, engaging, and perfectly executed frontend and backend experiences. While finding time to write, hike and read.

Blog

Your Cron Job Runs on Every Server

  • laravel
  • typescript
  • scheduling
  • architecture

Scheduled jobs are the easiest code in the app to get right on one server.

You write the command, add a line to the scheduler, and every night at 2am the invoices go out. For months. Nobody thinks about it.

Then traffic picks up and someone adds a second app server. Or the app moves to containers and the platform runs three copies. Nothing about the job changed. The next morning every customer has three invoices.

Every copy of the app has a clock

Laravel's scheduler works off one cron entry per server:

* * * * * cd /var/www/app && php artisan schedule:run >> /dev/null 2>&1

Every minute, schedule:run looks at what's due and runs it. That's the whole trick, and it's a good one.

But it's per server. If your deploy script installs that crontab on every box, every box runs schedule:run, and every box decides the 2am invoice job is due. Each one is correct. Together they're a bug.

Node apps hit the same thing with a library like node-cron or a plain setInterval in the server process. It feels like "the app" runs the job. Really, every process runs it. Scale to three and you get three.

Option one: only one thing runs the schedule

The simplest fix is structural. Pick one place that runs scheduled work and keep it off everything else.

That can be a dedicated worker server, or one container that runs php artisan schedule:work while the web containers don't. No locks, no cache dependency, very little to explain to the next person.

The catch is that "only one" has to stay true. Someone clones the server image, or bumps the scheduler container's replica count to two because "more is safer", and you're back to duplicates. Write it down where the infrastructure is defined, not just in your head.

Option two: let them all try, and let one win

If every server has to run the scheduler (or you can't control that), Laravel has onOneServer():

use Illuminate\Support\Facades\Schedule;
 
Schedule::command('invoices:send')
    ->dailyAt('02:00')
    ->onOneServer();

The first server to grab the task takes an atomic lock in the cache. The others see the lock and skip it.

Two requirements that are easy to miss. The default cache driver has to be one that supports this (database, memcached, dynamodb, or redis), and every server has to point at the same cache. If each box has its own local Redis, each box gets its own lock, which means no lock at all.

Closures need a name, because Laravel needs something stable to build the lock key from:

Schedule::call(fn () => Report::rebuildDaily())
    ->daily()
    ->name('rebuild-daily-report')
    ->onOneServer();

Overlap is a different problem

onOneServer() stops two servers from running the same task at the same time slot. It doesn't stop a slow run from bumping into the next one.

If a sync is scheduled every five minutes and one run takes eight, the next run starts while the first is still going. Same server, same job, two copies touching the same rows.

Schedule::command('inventory:sync')
    ->everyFiveMinutes()
    ->withoutOverlapping(10)
    ->onOneServer();

That 10 matters. By default the overlap lock lasts 24 hours. If the process gets killed halfway through (deploy, out of memory, server reboot), the lock can stay behind and the task quietly skips every run until it expires. Set the expiry to something a little longer than a normal run.

If you're already in that state, php artisan schedule:clear-cache clears the stuck scheduler locks.

Queued jobs need the same thinking

A lot of scheduled tasks just dispatch jobs. Those jobs can still overlap once they're on the queue, especially with several workers.

Laravel has job middleware for that:

use Illuminate\Queue\Middleware\WithoutOverlapping;
 
public function middleware(): array
{
    return [
        (new WithoutOverlapping($this->account->id))->expireAfter(300),
    ];
}

Keying it by account means two syncs for different accounts can run side by side, but two for the same account can't. That's usually the lock you actually want. "Only one of these globally" is often too blunt.

The Node version: let the database decide

If you're running a Node service on several instances and Postgres is already there, an advisory lock is a clean way to pick a winner:

import { Pool } from "pg";
 
const pool = new Pool();
const SEND_INVOICES_LOCK = 42_001;
 
export async function runExclusive(lockId: number, task: () => Promise<void>) {
  const client = await pool.connect();
  try {
    const { rows } = await client.query(
      "SELECT pg_try_advisory_lock($1) AS locked",
      [lockId],
    );
    if (!rows[0].locked) return; // another instance has it
 
    try {
      await task();
    } finally {
      await client.query("SELECT pg_advisory_unlock($1)", [lockId]);
    }
  } finally {
    client.release();
  }
}
 
// in the scheduled handler on every instance
await runExclusive(SEND_INVOICES_LOCK, sendInvoices);

The detail that bites people: advisory locks belong to the database session. You have to lock and unlock on the same connection. If you call pool.query() for the lock and pool.query() again for the unlock, the pool can hand you two different connections, and the unlock does nothing. That's why this checks out one client and holds it for the whole run.

The upside of session locks is that if the process dies, the connection drops and Postgres releases the lock for you. No 24 hour ghost.

Locks are the seatbelt, not the brakes

All of these locks reduce duplicates. None of them make duplicates impossible.

A lock can expire while a slow job is still running. A cache can get flushed. Someone can run the command by hand while the scheduled one is going. So the job itself should still be safe to run twice:

Invoice::query()
    ->where('status', 'ready')
    ->whereNull('sent_at')
    ->lazyById()
    ->each(function (Invoice $invoice) {
        $claimed = Invoice::whereKey($invoice->id)
            ->whereNull('sent_at')
            ->update(['sent_at' => now()]);
 
        if ($claimed) {
            $invoice->sendToCustomer();
        }
    });

The conditional update means only one run can claim each invoice, whether there's a lock or not. (lazyById() is there on purpose. Plain each() pages with offsets, and since this loop changes the column it filters on, offset paging would skip rows.) The lock keeps things tidy. The claim keeps customers from getting three emails.

The tradeoff is that the claim happens before the send. If the send fails, that invoice needs a retry path (a failed status, a log line someone actually reads). One missed email is still an easier morning than three duplicate ones.

Ask this before you scale out

Before adding a server, a replica, or an autoscaling rule, list the scheduled tasks and ask one question about each: what happens if this runs three times at once?

If the answer is "nothing, it's idempotent", great. If the answer is "three invoices", fix it while it's still a one server problem.

Your cron job doesn't know how many copies of your app exist. You have to tell it.

← All posts