Replaying Missed Webhook Events

When a webhook endpoint is unavailable for longer than the provider's retry window, those events are gone from the provider's queue and your suppression list is quietly wrong. This guide covers detecting the gap, recovering the window from the provider's API, and replaying it safely through the consumer you already have.

Root Cause: Retry Windows Are Shorter Than Outages

Providers retry a failed webhook delivery on a decaying schedule and then stop. The exact window varies — commonly a few hours to about a day — and it is always finite. An endpoint down for longer than the window loses every event whose retries expired during the outage.

The loss is silent in a specific and dangerous way. Delivery events going missing is inconvenient; bounce and complaint events going missing is a reputation problem, because those events drive suppression. An address that hard-bounced during the outage stays on your list, gets mailed again, bounces again, and contributes to the bounce rate that determines whether your account stays in good standing.

The gap is also invisible from inside your system. Nothing arrives, so no error is logged and no metric moves — a webhook consumer that receives nothing looks identical to one whose provider had nothing to send.

Only the middle-right band needs recovery, and identifying its boundaries precisely is what keeps the replay small.

Detecting the Gap

Because absence produces no signal, detection has to be positive: assert that events are arriving rather than waiting for a failure.

# gap_detector.py — a heartbeat over the event stream. Silence is the alarm.
from datetime import datetime, timedelta, timezone

# Below this rate, something is wrong even at quiet hours — tune to your floor.
MIN_EVENTS_PER_HOUR = 5


def check_stream(store, now=None):
    now = now or datetime.now(timezone.utc)
    last_hour = store.count_events(since=now - timedelta(hours=1))

    if last_hour < MIN_EVENTS_PER_HOUR:
        # Distinguish "we sent nothing" from "we received nothing", or a quiet
        # night looks identical to a broken endpoint.
        sent = store.count_sends(since=now - timedelta(hours=2))
        if sent > 0:
            return ("page", f"{sent} messages sent, only {last_hour} events received")
    return None


def find_gap(store, window_hours=72):
    """Return the (start, end) of the longest silence in the recent window."""
    stamps = store.event_timestamps(since=datetime.now(timezone.utc)
                                    - timedelta(hours=window_hours))
    if len(stamps) < 2:
        return None
    widest = max(zip(stamps, stamps[1:]), key=lambda p: p[1] - p[0])
    # A gap shorter than an hour is normal traffic variation, not an outage.
    return widest if (widest[1] - widest[0]) > timedelta(hours=1) else None

Comparing events received against messages sent is the check that works. Event volume alone falls overnight and rises in the morning, so an absolute threshold either alarms every night or misses a daytime outage. The ratio is stable regardless of traffic shape.

Recovering the Window

Every major provider exposes an API for querying message events over a time range, which is the source for the replay. The shape differs but the sequence does not: query the gap window, page through the results, and hand each event to the same consumer the webhook uses.

# replay.py — pull the gap window and feed it through the existing consumer.
def replay_window(provider, consumer, start, end, page_size=500):
    stats = {"fetched": 0, "applied": 0, "duplicate": 0, "failed": 0}
    cursor = None

    while True:
        page = provider.list_events(start=start, end=end,
                                    limit=page_size, cursor=cursor)
        for event in page.events:
            stats["fetched"] += 1
            try:
                # The SAME consumer, with the same idempotency key handling.
                # A replay must not take a different code path from live
                # delivery, or the two can diverge in exactly the situation
                # where correctness matters most.
                result = consumer.handle(event)
                stats["duplicate" if result.deduplicated else "applied"] += 1
            except Exception:
                stats["failed"] += 1
                # Keep going: one malformed event must not abandon the window.
        if not page.next_cursor:
            break
        cursor = page.next_cursor

    return stats

Routing the replay through the same consumer is the single most important decision. A separate replay path that writes directly to the suppression store will, sooner or later, apply a rule the live consumer does not — and the discrepancy surfaces months later as an address that should have been suppressed and was not.

The high duplicate count you will see is expected and reassuring: the window overlaps events that did arrive, and the idempotent consumer recognises them. A replay reporting zero duplicates means the window boundaries are wrong.

Convergence on one consumer is what guarantees a replayed bounce is treated exactly like a live one.

Verifying the State Converged

A replay that ran without errors is not evidence that the state is correct. Three checks confirm it.

Count reconciliation. Compare the number of terminal events in your store for the window against the provider's own count for the same window. A remaining discrepancy means the window was too narrow or a page was dropped.

Suppression spot-check. Take a sample of addresses that bounced during the window according to the provider, and confirm each is now suppressed. This tests the path end to end rather than the fetch alone.

Send-eligibility sweep. Query for addresses with a terminal event in the window that are still marked sendable. In a converged system that set is empty, and anything in it is a message that would have been sent to a dead address.

Run all three. The first catches a fetch problem, the second a consumer problem, and the third a state problem — and each can occur without the others.

Variant: Widening the Window Deliberately

When the gap boundaries are uncertain, replay a window wider than the outage rather than trying to be precise. Idempotency makes over-fetching cheap: extra events are recognised as duplicates and discarded, at the cost of some API calls and processing time.

The asymmetry justifies it. A window an hour too wide costs a few thousand redundant lookups; a window an hour too narrow leaves bounced addresses on the list indefinitely. Where the provider's API allows it, replaying a full day around a suspected outage is a reasonable default.

What the Provider's API Gives You, and What It Does Not

Replaying from an API is not quite the same as receiving the original webhooks, and three differences matter enough to design around.

The event shape can differ. Several providers return a slightly different structure from their query API than from their webhook payload — different field names, a different nesting, timestamps in a different format. A consumer written against the webhook shape will reject API-shaped events, so the replay needs a normalisation step that maps both into one internal representation. Doing that mapping at the edge, before the consumer, is what keeps the consumer itself single-path.

Not every event type is queryable. Delivery, bounce and complaint events are almost always available; engagement events such as opens and clicks are sometimes not, or are aggregated rather than itemised. Since the terminal events are the ones that matter for suppression, this is usually acceptable — but it means a replay does not fully restore the record, and any downstream analytics that assumes completeness will be subtly wrong for that window.

Ordering is not guaranteed. A query returns events in the API's order, which may be by creation time, by message id, or unspecified. Since the consumer already handles out-of-order arrival by comparing event timestamps rather than arrival order, this costs nothing — but a consumer that relied on webhook ordering will corrupt state during a replay, which is one more reason the two paths must be the same.

The practical consequence is that the normalisation layer is the piece worth testing carefully. It is small, it runs rarely, and it is the only new code a replay introduces — which makes it exactly the code most likely to be wrong when it finally runs.

The normalisation layer is the only new code a replay introduces, which makes it the piece most worth testing before you need it.

Pipeline Integration

Keep the replay as a maintained, tested command rather than a script written during an incident. The moment you need it is the moment you least want to be writing pagination logic, and a command with a rehearsed dry-run mode turns a stressful recovery into a routine one.

Give it a dry-run that reports what would change without writing, and run that first. It is also the cheapest way to validate the window boundaries: a dry run reporting almost all duplicates confirms the window covers the gap.

Validation and Deployment Checklist

Frequently Asked Questions

How far back can we replay?

As far as the provider retains events, which is commonly around 30 days and sometimes as little as a week. That retention is the real limit on recovery, and it is worth checking before you need it — a provider retaining seven days means an outage discovered after a fortnight is unrecoverable from their side.

Can we avoid replay entirely by queueing at the edge?

Largely, yes. An endpoint that does nothing but verify the signature and push the raw body onto a durable queue fails only if the queue is unavailable, which is a much smaller surface than a full consumer. That is the fast-acknowledgement pattern, and it converts most consumer outages into queue depth rather than loss.

What if the provider has no event API?

Then the endpoint's availability is your only protection, and the fast-acknowledgement pattern becomes mandatory rather than advisable. It is also worth weighing in provider selection: an ESP with no way to recover missed events is one where every endpoint outage is permanent data loss.

Should replayed events trigger downstream side effects?

Only the idempotent ones. Updating suppression state is safe and necessary; sending a customer a notification about a bounce that happened three days ago is not. Where the consumer triggers outward-facing actions, it needs an age check so a replay updates state without re-firing time-sensitive effects.

Webhook Event Pipelines — the consumer architecture the replay reuses
Building an Idempotent Webhook Consumer — what makes over-fetching safe
Bounce and Complaint Handling — the suppression state a gap corrupts
Email Deliverability and Reputation — the bounce rate an unrecovered gap inflates

← Back to Webhook Event Pipelines