Josh Davies

Senior Software Engineer

I build beautiful, engaging, and perfectly executed frontend and backend experiences. While finding time to write, hike and read.

Blog

Retrying Harder Won't Fix a 400

  • software-engineering
  • laravel
  • typescript
  • reliability

A payment provider returns 503. Your job retries. Good.

A payment provider returns 400 because the card number is missing a digit. Your job retries. Then retries again. Then fills the queue with the same bad payload until someone notices the noise.

Retries feel like resilience. Blind retries are just load you chose to add.

Not every failure wants another try

A retry only helps when the next attempt might succeed without you changing the request.

Worth retrying:

  • Timeouts
  • Connection resets
  • 429 with a retry-after
  • 502, 503, 504 from a dependency that was briefly overloaded

Not worth retrying:

  • 400 and 422 (your payload is wrong)
  • 401 and 403 (auth won't magically heal mid-loop)
  • 404 on a resource that doesn't exist
  • Most 4xx, honestly

If the answer won't change until a human fixes the input, stop. Log it. Fail the job. Don't burn attempts proving the same fact.

Laravel makes retrying easy. Make the filter easy too.

Http::retry() is fine. The important part is the when callback, not the attempt count.

$response = Http::retry(
    times: 3,
    sleepMilliseconds: 200,
    when: function ($exception, $request) {
        if ($exception instanceof ConnectionException) {
            return true;
        }
 
        if ($exception instanceof RequestException) {
            $status = $exception->response?->status();
 
            return in_array($status, [429, 502, 503, 504], true);
        }
 
        return false;
    },
    throw: true,
)->post($url, $payload);

Three tries. Short sleep. Only the statuses that might clear on their own. Everything else fails once and tells you the truth.

Leave when off and Laravel retries any failed request, 400s included. That's the default you want to override. And be deliberate about 429. Rate limits need space, not enthusiasm.

Backoff is not optional

Three retries with no delay is three punches at the same bruised service.

Space them out, and add a little randomness so every worker doesn't wake up on the same millisecond:

async function withRetry<T>(
  fn: () => Promise<T>,
  opts: { attempts?: number; baseMs?: number } = {},
): Promise<T> {
  const attempts = opts.attempts ?? 3;
  const baseMs = opts.baseMs ?? 200;
  let lastError: unknown;
 
  for (let i = 0; i < attempts; i++) {
    try {
      return await fn();
    } catch (error) {
      lastError = error;
      if (!isTransient(error) || i === attempts - 1) {
        throw error;
      }
 
      const jitter = Math.floor(Math.random() * baseMs);
      const delay = baseMs * 2 ** i + jitter;
      await new Promise((r) => setTimeout(r, delay));
    }
  }
 
  throw lastError;
}
 
// HttpError is your own error class that carries the response status.
function isTransient(error: unknown): boolean {
  if (!(error instanceof HttpError)) {
    return error instanceof TypeError; // failed to fetch, DNS, etc.
  }
 
  return [429, 502, 503, 504].includes(error.status);
}

200, 400, 800 plus jitter beats 0, 0, 0. Your dependency notices the difference, and so does your error budget.

A sick service plus retries is how you finish it off

The dependency is slow. Your timeouts fire. Fifty workers retry at once. Now it's not slow. It's dead, and you helped.

A few habits that keep you from joining the pile-on:

  • Cap attempts. Three is usually enough. Ten is a confession.
  • Honor Retry-After when you get a 429.
  • Prefer a queue with delay over a tight loop in a web request. The user shouldn't wait through your backoff math.
  • If the call isn't safe to run twice, fix that first. Retrying a non-idempotent charge is a finance meeting, not a reliability win.

Fail loud when you're done

Retries without a final failure just hide the outage until the queue is a thousand jobs deep.

When the last attempt fails, throw. Surface the status code, the request id if you have one, and which attempt died. Then let the job fail, alert, or dead-letter like you meant it.

A single clear failure is useful. A quiet retry loop that "almost works" is how Sunday night starts.

The habit

Before you add a retry, answer three questions:

  1. Which failures get another try, and which ones don't?
  2. How long do you wait between attempts, and do the workers desync?
  3. What happens after the last attempt?

If the plan is "just keep trying," you don't have retries. You have a denial of service pointed at yourself.

← All posts