Why the same event arrives twice
Duplicate delivery is not one bug with one fix. It is four independent mechanisms that happen to produce the same symptom, and a handler has to survive all of them.
Your endpoint committed the change, then timed out. You wrote the payment to
your database, then spent another few seconds sending a receipt, and the
provider gave up waiting before your 200 arrived. From the provider’s side
that delivery failed. From your side it succeeded completely. The retry is
correct behaviour on both sides.
The provider retried after a genuine failure. Your service was redeploying, the database connection pool was exhausted, a dependency was down. Some of those failures happen after a partial write.
Somebody pressed resend. Support staff and engineers replay events from a provider dashboard when investigating a discrepancy. That replay is indistinguishable from a retry at your endpoint, and it can arrive days after the original.
Your consumer restarted mid-processing. With an at-least-once queue between the endpoint and the business logic, an unacknowledged message is redelivered after a crash — a duplicate your provider never sent and never sees.
Only the second of these is a failure in the ordinary sense. The other three happen while everything is working as designed, which is why “we will fix the timeouts” is not a strategy.
Four ways this shows up in production
- The ledger double-posts. Two credits for one payment. The books stop agreeing with the provider’s settlement report, and nobody notices until reconciliation.
- The customer gets two confirmation emails. Cheap, visible, and the fastest way to find out you have the problem at all.
- Stock is decremented twice. One sale removes two units, and the discrepancy is discovered at stocktake rather than at the point of failure.
- A digital product is fulfilled twice. Two licence keys, two downloads, two entitlements — usually irreversible and occasionally expensive.
Note how the severity ranking is not the visibility ranking. The email is harmless and obvious; the ledger entry is costly and silent.
Handling it correctly
The rule is short: deduplicate inside the same transaction that changes business state. Anything else leaves a window.
Use a unique constraint, not a read-then-write check. The intuitive implementation asks the database whether the event was already processed and inserts a record if it was not. Two deliveries handled at the same time — by two workers, or by a retry that overlaps a slow first attempt — both read “absent” before either writes, and both proceed. A unique constraint on the deduplication key moves the decision into the database, where exactly one writer wins and the other gets a violation it can safely treat as “already done”. Insert that row in the same transaction as the ledger entry; if they can commit separately, they eventually will.
Choose the key carefully. Logging the provider’s event ID is the obvious first move and it is correct for the plain case: the same event redelivered carries the same ID. But it is not sufficient on its own. Some providers emit more than one event object for a single underlying occurrence, each with its own event ID — deduplicating on the ID alone lets both through, because to the key they are different events. A key built from the identifier of the object the event is about, together with the event type, collapses those correctly. The provider panel below records where each provider documents this.
Return the success status before the slow work. Acknowledge, then process asynchronously. This shrinks the timeout-retry mechanism, which is the most common source of duplicates, though it does not remove the need for idempotency — it just stops you manufacturing extra duplicates yourself.
A related hazard is worth naming so it is not confused with this one: several different events can be generated for one business transition, and they are not guaranteed to arrive in the order they were created. Those are not duplicates and deduplication will not help; they need a handler that does not depend on arrival order.
Testing it, and why a naive test passes
This is where most integrations get false confidence.
A duplicate delivery that a provider actually sends reuses the exact same raw body and the same event ID, and carries a freshly computed, independently valid signature with its own timestamp. Both properties matter, and a test that gets either one wrong is worse than no test, because it reports success.
Two common ways to get it wrong:
- The application deduplicates on the signature header. Since each delivery is signed independently, the header differs between the original and the duplicate. Deduplicating on it means never recognising a duplicate at all — and a test harness that replays a recorded request byte for byte, signature included, will never reveal this, because in that test the headers do match.
- The harness replays a recorded request verbatim. This exercises a case the provider never produces. Depending on the signature tolerance window, it either passes trivially or fails for a reason unrelated to idempotency. Either way, it tells you nothing about the behaviour you care about.
The scenario below delivers the same event twice the way a provider does: one immutable body and event ID, one fresh signature per attempt.
What this scenario does not prove
The duplicate is delivered sequentially — the second request is sent only after the first has completed and returned a success status. That is the common real-world shape: a retry after your handler timed out arrives seconds or minutes later, not concurrently. It is not the hard case.
Two deliveries processed in parallel, by two workers or by a retry overlapping a slow handler, can both pass a read-then-write duplicate check and both commit. A deduplication built on a unique constraint survives that; one built on “select, then insert if absent” does not, and this scenario will not tell you which one you have.
One delivery count, one event count
The invariant worth carrying into your own tests: duplicated delivery must not create a second logical event. Delivery count and event count are separate assertions and should be asserted separately. An integration that records two payment events because it received two deliveries has already lost — the downstream damage is just a matter of which system reads that record first.