Josh Davies

Senior Software Engineer

I build beautiful, engaging, and perfectly executed frontend and backend experiences. While finding time to write, hike and read.

Blog

Let It Finish Before You Kill It

  • devops
  • laravel
  • php
  • typescript
  • node

Every deploy ends a bunch of processes. Web servers, queue workers, the Node service somebody wrote in a weekend. Something has to tell them to stop.

That something is almost always a signal. SIGTERM first, which politely asks the process to wrap up. Then, after a timeout, SIGKILL, which does not ask.

If your app does nothing with the first one, you get the second one. Whatever was in flight at that moment just stops. Half a request. Half an import. A charge that went through with no record saved.

The timeout is shorter than you think

Each layer gives you a grace period between the polite signal and the kill:

  • docker stop waits 10 seconds by default.
  • Kubernetes waits 30 seconds by default (terminationGracePeriodSeconds).
  • Supervisor waits 10 seconds by default (stopwaitsecs).

Ten seconds is fine for a web request. It is not fine for a queue job that builds a report for four minutes. That job is getting killed on every deploy, and nobody notices until a customer asks where their export went.

Laravel workers already know how to do this

php artisan queue:work listens for SIGTERM. When it gets one, it finishes the job it is on and then exits. That is exactly what you want.

Two things quietly break it.

First, the signal handling needs the pcntl extension. No pcntl, no graceful stop. Check your production image, not just your laptop:

php -m | grep pcntl

Second, the grace period has to be longer than your longest job. Laravel's docs say this directly for Supervisor. If a job can run for five minutes, ten seconds of stopwaitsecs means it gets killed:

[program:worker]
command=php /var/www/artisan queue:work --timeout=300
stopwaitsecs=330

Same idea in Kubernetes. Raise terminationGracePeriodSeconds for the worker deployment, not just the web one.

And while you are in there, keep retry_after in config/queue.php longer than the job's --timeout. Otherwise the queue thinks the job died and hands it to another worker while the first one is still running it.

For normal deploys, php artisan queue:restart is the gentle version of all this. Workers finish their current job, exit, and Supervisor starts them again on the new code.

Node will not do it for you

A plain Node HTTP server has no idea a deploy is happening. You have to wire it up:

import http from "node:http";
 
const server = http.createServer(app);
server.listen(3000);
 
let shuttingDown = false;
 
process.on("SIGTERM", () => {
  shuttingDown = true;
 
  // Give the load balancer a few seconds to see the failing health check.
  setTimeout(() => {
    server.close(() => process.exit(0));
    server.closeIdleConnections();
  }, 5_000);
 
  // Don't wait forever. Exit before the platform kills us.
  setTimeout(() => process.exit(1), 25_000).unref();
});

The shuttingDown flag lets your health check start returning a 503, which tells the load balancer to stop sending you new traffic. The short delay before closing gives it time to notice. Close right away and requests that were already headed your way hit a closed port.

After that, server.close() stops accepting new connections and waits for open requests to finish. closeIdleConnections() drops keep-alive connections that are just sitting there, so they don't hold the process open. The last timer makes sure you leave on your own terms before the grace period runs out.

The health check itself is one line:

app.get("/health", (req, res) => {
  res.sendStatus(shuttingDown ? 503 : 200);
});

If you run queue workers in Node, most libraries have a close method that waits for active jobs. With BullMQ it looks like this:

process.on("SIGTERM", async () => {
  await worker.close(); // waits for the current jobs to finish
  process.exit(0);
});

Make sure the signal actually arrives

This one is sneaky. You can write a perfect handler and never see it run.

In a Dockerfile, the shell form of CMD runs your app under /bin/sh -c, so the shell can end up being the process that receives the signal:

# The shell may get SIGTERM instead of your app.
CMD node dist/server.js
 
# Your app is the process that gets the signal.
CMD ["node", "dist/server.js"]

Wrapping things in npm start or a shell script can cause the same problem, because the wrapper is the process receiving the signal and it may not pass it along. Run the real process directly when you can.

There is also the PID 1 problem. The process running as PID 1 in a container doesn't get the default signal behavior, so a process that would normally die on SIGTERM just ignores it. Docker sits there for 10 seconds and then kills it. docker run --init (or init: true in Compose) puts a tiny init process in front that forwards signals properly.

A quick way to test all of this: start the container, send it a stop, and time it.

time docker stop my-app

If it takes almost exactly 10 seconds, your app never handled the signal. It got killed.

You will still get killed sometimes

Graceful shutdown covers deploys and scale-downs. It does not cover the server losing power, the OOM killer, or a job that ignores the timeout.

So the other half is making jobs survive being cut off:

  • Break long jobs into smaller ones, or save progress as you go so a retry picks up where it left off.
  • Make jobs safe to run twice, because a killed job is going to get retried.
  • Do the side effect (the charge, the email) last, after the work that might fail.

Graceful shutdown makes interruptions rare. Idempotent jobs make them harmless. You want both.

The short version

  • Deploys send SIGTERM, then SIGKILL after a timeout. Handle the first one.
  • Laravel workers stop gracefully, but only with pcntl and a grace period longer than your longest job.
  • Keep retry_after longer than the job timeout.
  • In Node, fail the health check, then close the server, drain connections, and exit before the deadline.
  • Use the exec form of CMD, skip wrappers that swallow signals, and use --init in containers.
  • Time docker stop. Ten seconds flat means nobody was listening.

Your process gets a warning before it dies. Use it.

← All posts