Making payment webhook handling idempotent
Customers were debited and still showing as unpaid, because the webhook handler was racing against the provider's timeout.
The symptom
Customers were successfully charged by the payment provider, but our system still showed them as unpaid — which meant support tickets, and a growing pile of manual reconciliation.
Root cause
The webhook handler processed the full payload synchronously — validating it, updating records, sending confirmation emails — before sending back an acknowledgment. Under any load, or with a slow downstream call, that took long enough that the provider's timeout fired first and dropped the connection before the acknowledgment landed.
From the provider's side, that read as a failed delivery — which meant retries, sometimes arriving after the original request had already partially applied.
The fix
The webhook endpoint now acknowledges immediately on receipt, and hands the payload off to a queue for asynchronous processing. That alone fixes the timeout problem.
It doesn't fix duplicate delivery on its own, though — providers retry webhooks whether or not the first one actually failed. So the queue handlers are written to be idempotent: processing the same payload twice has to produce the same end state as processing it once, not a double-applied payment.
Takeaway
Webhooks will be delivered more than once, whether you plan for it or not. The only handler design that's actually safe is one that assumes duplicate delivery from the start.