Why delivery order breaks
Three separate mechanisms can reorder webhook events relative to when they actually occurred, and a consumer has to survive all three, not just the one that is easiest to picture.
Parallel dispatch on the provider’s side. A provider’s own delivery infrastructure is not a single queue emptied one event at a time; it is typically a fan-out across many workers, and nothing about that architecture guarantees the order events leave it matches the order they were generated in.
A retried older event overtaking a newer one. If the first delivery attempt for an earlier event is slow, fails, or is queued for retry, while a later event sails through on its first attempt, the later event can land at your endpoint before the earlier one’s retry does — even though the provider generated them in the correct order.
Concurrent consumers on your own side — the one most readers have not considered. Even a provider that delivered every event in perfect order does not help you if your own infrastructure processes deliveries with a pool of workers. Two adjacent deliveries picked up by two different workers finish in whatever order database contention, garbage collection, or plain scheduling luck produces, and that has nothing to do with the webhook delivery itself. An integration that carefully verifies the provider’s ordering guarantees and then hands every delivery to a worker pool has reintroduced the exact problem it thought it had ruled out.
Why sorting by timestamp does not fix it
The intuitive fix — buffer incoming events for a short window and apply them in timestamp order — fails for two independent reasons.
First, it requires an unbounded wait to be actually correct. Any fixed buffering window is a bet that no event will ever arrive later than that window, and the moment one does, you are back to applying an out-of-order update, just with extra latency added to every single event to make it rarer.
Second, it still loses to clock skew. The timestamp on an event is generated on the provider’s clock; the order that actually matters to your application is the order state genuinely changed, and comparing a provider timestamp against your own consumer’s notion of time near the margins is exactly the kind of comparison that is unreliable by construction, not merely difficult to get precisely right.
The correct fix: state versioning
Apply an event only when the state version it carries is newer than the version you already have stored for that resource, and drop a stale event instead of replaying or reprocessing it. This is not a workaround invented for this article — it is the mechanism the canonical model already uses.
Every canonical resource snapshot — a payment, a payment_attempt, and so on
— carries a monotonically increasing stateVersion (ADR 0008). Every canonical
event that reports on a resource carries resourceStateVersion, which is
exactly the stateVersion of the resource snapshot that event is about. A
consumer that stores, alongside each resource it tracks, the highest
resourceStateVersion it has successfully applied has everything it needs: on
each incoming event, compare the event’s resourceStateVersion against the
stored value for that resource, apply the event and update the stored value only
if the event’s version is strictly greater, and otherwise drop the event as
stale.
This rule is correct under arbitrary ordering and arbitrary delay because it never asks “did this arrive before or after some other event” — a question that delivery order can answer wrongly. It only asks “is this newer than what I already have,” which delivery order cannot affect at all.
Consider a two-stage payment, where
authorization and capture are separate
operations: an authorization succeeding produces payment.authorized at
resourceStateVersion: 2, and the later capture produces payment.succeeded at
resourceStateVersion: 3, both against the same payment resource. Suppose your
consumer receives payment.succeeded (version 3) first — reversed relative to
when the two transitions actually happened — followed by payment.authorized
(version 2). State versioning gets this right without knowing anything happened
out of order: it applies version 3 because it is newer than the version 0
baseline, stores 3, then receives version 2, finds that 2 is not newer than the
3 already stored, and drops it. The payment correctly ends up recorded as
succeeded either way — the mechanism does not care which event arrived first,
which is exactly the property a correct handler needs.
The late webhook
An event can arrive long after the transition it describes — minutes, hours, or,
if it was retried against a temporarily unreachable endpoint, considerably
longer. “We already moved on” is not by itself a reason to ignore it: lateness
and staleness are different properties, and only staleness — a
resourceStateVersion that is not newer than what you already have — is a valid
reason to drop an event. A late event that is still the newest version you have
seen for that resource carries real information your stored state does not yet
have, and dropping it purely because time has passed leaves your application
permanently behind the provider on that resource. The state-versioning rule
above already gives the correct answer in both cases without needing a separate
notion of “too late” at all.
The classic race: a webhook faster than your own write
The most common production incident in this class is not a reordering across several events — it is a single webhook arriving before the local transaction that created the record it refers to has committed. A typical sequence: your application creates an order row, calls the provider’s API to start the payment, and is still waiting on that call (or on a redirect, or on its own transaction committing) when the provider’s webhook — sometimes genuinely delivered within milliseconds — reaches your endpoint and looks up an order that, from that request’s point of view, does not exist yet.
The accepted handling is to treat “the referenced record does not exist yet” as a retryable condition, not a permanent failure: respond with a status your provider’s retry contract treats as “try again,” and let the provider’s normal at-least-once retry behavior give your own write the time it needs to land. Treating a not-yet-visible record as a hard failure drops real events; racing to create the record from inside the webhook handler itself risks a second, conflicting write once your original transaction does commit.
What you can run today, and what is coming
The scenarios that would let you run the exact failures above — a reordering scenario, a stale event delivered late, a delay across a consumer restart — are planned but not yet implemented. Reversed order and delayed delivery are not something Finxture’s extension can produce on demand today: they depend on a scheduling capability the engine does not have yet, and this page will be updated to link to them, rather than silently gain scenario cards, once they ship.
What is available today is duplicate webhook delivery — a different at-least-once delivery problem with its own runnable scenario — and the state-versioning approach itself, which you can and should build against the canonical model described above regardless of whether an authored scenario can yet reproduce the failure it protects you from.
What to test, once these scenarios ship
- Reversed order for a two-stage payment — deliver the capture-stage events before the authorization-stage events for the same payment, and confirm the stored state still ends up correct.
- A stale event delivered after a newer one — redeliver an already-superseded event for a resource your consumer has already moved past, and confirm it is dropped rather than reapplied.
- A long delay across a consumer restart — confirm the “highest version applied” state survives a process restart, since a version comparison kept only in memory silently reverts to accepting anything the moment the process restarts.