Josh Davies

Senior Software Engineer

I build beautiful, engaging, and perfectly executed frontend and backend experiences. While finding time to write, hike and read.

Blog

Treat Every Integration Like It Will Break

  • software-engineering
  • architecture
  • apis

Every app that does real work eventually talks to someone else’s system.

Payments. Email. Shipping rates. Tax. CRM sync. A webhook that is supposed to tell you when an order changes. It works in staging. It works on launch day. Then a Tuesday arrives and the remote API times out, returns a half payload, or quietly changes a field name.

If your feature assumes the other side is always healthy, you didn’t finish the feature.

The happy path is the easy part

Calling an API is usually a few lines.

Send the request. Decode the JSON. Save what you need. Move on.

That code is fine for a demo. Production needs the boring questions answered up front: What if it hangs? What if it returns 500? What if it returns 200 with a body that doesn’t match the docs? What if the same webhook arrives twice? What if it never arrives at all?

Those aren’t edge cases. They are the job.

Separate “we accepted it” from “they confirmed it”

A lot of integration bugs come from treating a remote call like a local database write.

Local writes are yours. You can wrap them in a transaction. You can roll them back. You know when they committed.

Remote calls are a negotiation. You asked. They might have done it. They might have done it and your response got lost. They might have refused. Your UI should not pretend those are the same outcome.

Save your own record in a clear state: pending, submitted, confirmed, failed. Update it when you actually know more. Show the user something honest while that state is in between.

Timeouts are a feature

An integration without a timeout is a way to freeze your workers for sport.

Pick a limit. Fail loudly when you hit it. Retry only when retrying is safe. Not every POST is safe to fire again just because the first response was slow.

Idempotency keys, remote request IDs, and “create or find” style endpoints exist for a reason. Use them when money, inventory, or duplicate emails are on the line.

Webhooks lie in both directions

Providers will say they deliver webhooks reliably. Sometimes they do. Sometimes they retry for hours. Sometimes they deliver the same event more than once. Sometimes your endpoint was down and you only find out from a dashboard three days later.

Verify signatures. Make handlers idempotent. Store the event id you’ve already processed. Do the minimum work in the HTTP response and push the rest to a queue when you can.

And have a backfill path. If the webhook is the only way your system learns the truth, you will eventually be wrong with confidence.

Log the conversation, not just the crash

When an integration fails, “Error syncing customer” is almost useless.

You want enough context to replay the argument: which local record, which remote id, which endpoint, which status code, which truncated body, how many retries, what you decided afterward.

Don’t log secrets. Do log enough that a human can tell whether you should retry, ignore, or call support with evidence.

Keep their shape out of your core

Map remote payloads at the boundary.

Your domain should talk in your terms: Order, Customer, Shipment. Not in whatever nested structure the vendor shipped this quarter. When they rename a field, you want one mapper to change, not twelve controllers and a report query.

Same idea for errors. Translate vendor noise into something your app and your users can understand.

What “done” looks like

An integration is done when you can answer yes to these:

We know what pending means. We know what success means. We know what failure means. We can retry without duplicating damage. We can explain a bad Tuesday from the logs. We can keep serving users while the other system is having a moment.

None of that is glamorous. It’s the difference between a demo that talks to Stripe and a product that still takes orders when Stripe is slow.

Treat every integration like it will break. Then write the small amount of extra code that makes that boring instead of exciting.

← All posts