Your job ran once. Then the worker crashed after the write but before the ack. Then the provider resent the webhook. Then the user mashed Submit because the spinner looked stuck.

Three more executions. Same intent. If your code treats every run like the first, you now have duplicate orders, duplicate emails, or a remote record created twice.

Retries are not optional in real systems. Duplicate-safe writes are.

What goes wrong

Most write paths are written for the happy once:

Create the row. Call the API. Send the email. Return 200.

That works until the second attempt lands. "Already done" and "never started" look the same if you only insert and hope. Race two requests and you get two inserts before either can check.

The fix is not "never retry." The fix is make the second run a no-op (or the same result) instead of a second side effect.

Laravel: unique keys beat hope

Give the work an identity the database can enforce.

For webhooks, store the delivery id with a unique index and bail if you have seen it:

public function handle(WebhookPayload $payload): void
{
    try {
        ProcessedWebhook::query()->create([
            'provider' => 'billing',
            'delivery_id' => $payload->deliveryId(),
        ]);
    } catch (UniqueConstraintViolationException $e) {
        return; // already handled
    }
 
    $this->apply($payload);
}

For domain writes, prefer upsert keyed on a natural unique, not "select then insert":

Subscription::query()->updateOrCreate(
    ['external_id' => $dto->externalId],
    ['status' => $dto->status, 'renews_at' => $dto->renewsAt],
);

Laravel's unique job middleware can also stop two workers from processing the same order_id at once. That helps under load. The database constraint is still the source of truth when a retry happens later.

If the side effect is an email or a remote create, record a local marker with the business write when you can. Retry then sees "already sent" and stops.

TypeScript: client keys and upserts

The same smell shows up in APIs and Node workers. A double-click on "Pay" should not mean two charges.

Accept an idempotency key, persist it uniquely, and reuse the first result:

async function createCheckout(input: {
  cartId: string;
  idempotencyKey: string;
}): Promise<Checkout> {
  const existing = await db.checkout.findUnique({
    where: { idempotencyKey: input.idempotencyKey },
  });
  if (existing) return existing;
 
  try {
    return await db.checkout.create({
      data: {
        cartId: input.cartId,
        idempotencyKey: input.idempotencyKey,
        status: "pending",
      },
    });
  } catch (e) {
    if (isUniqueViolation(e)) {
      return db.checkout.findUniqueOrThrow({
        where: { idempotencyKey: input.idempotencyKey },
      });
    }
    throw e;
  }
}

For sync-style updates, upsert on the natural key:

await db.customer.upsert({
  where: { externalId: payload.externalId },
  create: { externalId: payload.externalId, email: payload.email },
  update: { email: payload.email },
});

The unique index is doing the real work. Your code is just cooperating with it.

When you can skip the ceremony

Read-only paths do not need this. Pure caches you can rebuild do not either.

Internal tools where a duplicate is cheap and obvious can stay simple until they hurt.

Do not invent an idempotency key when you already have a natural unique, like external_id or order_id + event_type. Use the identity you have.

And do not confuse "at most once" delivery fantasies with a design. Networks lie. Assume at least once and make the handler safe.

The habit

Before you ship a write, a job, or a webhook handler, ask: if this runs twice, what breaks?

If the answer is money, inventory, email, or a remote create, add a unique key, an upsert, or a processed-event row before you add another retry.

Make it safe to run twice. Then retries stop being scary and start being boring.