Why Your Payment Webhook Handler Works in Testing and Breaks in Production
A team ships a Stripe integration, tests it against a handful of sandbox events, watches subscriptions activate correctly, and calls it done. Three weeks later, at real volume, support starts getting tickets: a customer was charged twice, another's subscription shows active in Stripe but not in the app, a refund never updated the internal ledger. Nobody touched the webhook code. It just stopped being reliable once it met production traffic patterns that the sandbox never produced.
This is one of the most common gaps in payment integrations, and it's rarely a Stripe problem. It's a webhook handler that was written to handle the happy path, one event, processed once, in order, with the endpoint always up. Production doesn't work that way, and payment processors are explicit that it won't: webhooks are delivered at-least-once, not exactly-once, and ordering is not guaranteed.
The four failure modes that actually happen
1. Duplicate delivery
Every major payment processor retries webhook delivery if it doesn't get a fast 2xx response, and it retries on a schedule that can span hours or days. If your endpoint is slow, times out, or returns a 500 for an unrelated reason, the same event arrives again, sometimes several times. A handler that isn't idempotent will process a charge.succeeded event twice and, depending on what it's wired to do, provision a resource twice, send a duplicate receipt, or double-decrement inventory.
The fix has to happen at the data layer, not just in application logic. Before processing an event, check whether its event ID has already been recorded as processed, inside the same transaction that performs the side effect. An in-memory check or a check-then-write with a gap between them will still race under concurrent delivery, which does happen when a processor fires a retry while the original request is still being handled.
2. Out-of-order events
Webhook events are not guaranteed to arrive in the order they were generated. A subscription.updated event can arrive before the subscription.created event it logically follows, especially under retry conditions where an earlier delivery failed and a later one succeeded first. A handler that assumes strict ordering, for example one that upserts a subscription record and trusts the latest webhook payload as the source of truth for current state, can end up overwriting newer state with older data.
The reliable pattern is to treat each event as reporting on a snapshot of the object at a point in time, and only apply it if it's newer than what's stored. Most processors include a timestamp or a monotonically increasing sequence you can compare against a last_updated field before applying the update, rather than blindly writing whatever the event says.
3. Silent handler failures
A webhook handler that catches an exception, logs it, and still returns a 200 looks fine in a dashboard and is actually the worst version of this bug, because the processor now believes delivery succeeded and will never retry. The event is gone. If that event was a failed charge notification, a subscription cancellation, or a dispute, the internal system state is now permanently wrong until someone notices manually, usually because a customer complains.
The fix is boring but non-negotiable: any exception during processing should result in a non-2xx response, so the processor's retry mechanism does its job. Application-level error handling and delivery-acknowledgment are two different concerns, and conflating them is what causes silent data loss.
4. Endpoint downtime during deploys
Webhook delivery doesn't pause because your service is mid-deploy. If a deployment causes even thirty seconds of 502s on the webhook endpoint, every event sent in that window either gets retried (fine, if the retry logic and idempotency are solid) or, in poorly configured setups, gets marked as permanently failed after a limited number of attempts and just stops. Teams that haven't stress-tested their deploy process against webhook traffic often don't discover this until a high-traffic release window overlaps with a burst of checkout activity.
Why testing against the sandbox doesn't catch any of this
Sandbox testing exercises the happy path by design: one event, clean payload, endpoint always responsive, no concurrent load. None of the four failure modes above are triggerable that way. They only show up under real concurrency, real network flakiness, and real retry timing, which is exactly why teams ship integrations that pass every test and still fail in production within weeks.
What a correctly built webhook layer actually looks like
Idempotency at the storage layer
Every incoming event ID gets checked against a durable store as part of the same database transaction that applies its side effects, not as a separate pre-check. If the event ID already exists, the handler returns success immediately without reprocessing.
A queue between receipt and processing
The webhook endpoint's only job should be to verify the signature, persist the raw event, and return a fast 2xx. Actual processing, provisioning resources, updating subscription state, sending notifications, happens asynchronously from a queue. This decouples "did we acknowledge the event" from "did we finish handling it," which means a slow downstream dependency can never cause a processor-side timeout and retry storm.
Ordering guards based on object state, not delivery order
Each update checks a version or timestamp field on the target object before applying, so a late-arriving stale event can't clobber newer state.
A dead-letter path for events that fail repeatedly
If an event fails processing after a reasonable number of attempts, it shouldn't vanish into logs. It needs to land somewhere a human can see it, replay it, and resolve it, otherwise the failure modes above turn into slow, invisible data corruption instead of loud, fixable errors.
Signature verification on every request
This one is basic but still gets skipped under deadline pressure: verifying the webhook signature using the processor's signing secret before trusting any payload. An unverified webhook endpoint is a public API that lets anyone simulate a successful payment.
Monitoring on event lag and failure rate
The gap between when an event was generated and when it was successfully processed is one of the highest-signal metrics for a payment integration, and almost nobody tracks it until after an incident.
Where this connects to the rest of the payment stack
Webhook reliability isn't an isolated concern, it's the nervous system for everything else in a payment integration. A Payment Webhook Architecture built with idempotency, ordering guards, and a proper dead-letter queue is what keeps a Stripe Payment Integration accurate under real load, and it's the same infrastructure that a dunning and recovery system depends on to know, correctly and immediately, when a retried charge actually succeeded. If the webhook layer is unreliable, every downstream system built on top of it, billing state, fraud checks, provisioning, inherits that unreliability whether it's visible yet or not.
A quick self-check
If you're not sure where your current integration stands, these five questions expose most of the risk:
- If the same event ID arrives twice, does anything break, or is it a guaranteed no-op?
- Does the handler ever return 200 after catching an exception internally?
- Is event processing synchronous inside the request handler, or queued?
- Is there a stored, queryable record of every event received, independent of whether processing succeeded?
- Has anyone actually simulated duplicate and out-of-order delivery in a test environment, rather than just single clean events?
Most teams can answer "no" to at least two or three of these on their first pass, and that's usually exactly where the next production incident is going to come from.