Skip to main content

Designing a Dead-Letter Queue for Failed Transactional Sends

Give failed email sends a terminal state: what belongs in a dead-letter queue, the payload it must retain, alerting on any arrival, and a safe replay path.

A message that fails deterministically will fail identically on every retry. Without a terminal state it circulates forever, consuming worker slots and rate-limit tokens that real sends need. This guide covers the dead-letter queue that ends that loop: what belongs in it, what each entry must carry, and how to replay one safely once the cause is fixed.

Root Cause: Retry Loops Have No Natural End

A retry policy answers "how long before trying again" but not "when to stop". Without an explicit bound, three things go wrong at once.

The message never leaves the queue, so queue depth stops being a useful signal — a queue holding a hundred permanently failing messages looks the same as one holding a hundred pending ones. Each retry consumes a rate-limit token that a deliverable message could have used, so a handful of poison messages measurably reduces throughput. And because nothing surfaces, the failure is invisible: no alert fires, and the user who never received their receipt is the first to notice.

The distinction that makes the bound possible is between failures where retrying could plausibly change the outcome and failures where it cannot. A 503 from the provider might succeed in thirty seconds. A 400 for a malformed recipient address will return 400 forever.

Failure classes and retry behaviour Transient failures are retried with backoff, ambiguous failures are retried with the same idempotency key, and permanent failures dead-letter immediately. Three Classes, Three Destinations transient 503, 504, connection reset Retry with backoff. Usually succeeds. counts against the budget ambiguous timeout after the request Retry with the same idempotency key. may already have been sent permanent 400, 404, 422 Dead-letter at once. No retry can help. alert immediately
The middle class is the one that needs care: a timeout may mean the message was sent, which is why the idempotency key must survive the retry.

The Exact Fix: What an Entry Must Carry

A dead-letter entry that records only "send failed" is close to useless — the person triaging it cannot tell what was meant to be sent, to whom, or why it stopped. Five fields make an entry actionable.

# dead_letter.py — the record written when a send reaches its terminal state.
from dataclasses import dataclass, field, asdict
import json


@dataclass
class DeadLetter:
    # 1. The complete original job, so a fixed message can be replayed rather
    #    than reconstructed from a log line.
    job: dict
    # 2. Why it stopped: the classification, not just the last error string.
    reason: str                    # "permanent_4xx" | "budget_exhausted" | "handler_error"
    # 3. The provider's own response, verbatim. Paraphrasing loses the detail
    #    that identifies the cause.
    last_error: str
    # 4. The full attempt history, so a slow-degrading failure is visible.
    attempts: list[dict] = field(default_factory=list)
    # 5. When it entered — required for both alerting and replay ordering.
    dead_lettered_at: str = ""

    def to_json(self) -> str:
        return json.dumps(asdict(self))


def dead_letter(store, job, reason: str, last_error: str, attempts: list[dict]) -> None:
    entry = DeadLetter(
        job=job,
        reason=reason,
        last_error=last_error[:2000],     # bound it; some providers return HTML
        attempts=attempts,
        dead_lettered_at=now_iso(),
    )
    store.append(entry.to_json())
    # Any arrival is an incident: this is a message a real user will not receive.
    metrics.increment("email.dead_letter", tags={"reason": reason})

Retaining the complete job rather than a reference to it is the decision that pays off during an incident. References go stale — the order is amended, the template is renamed, the user record is deleted — and a replay that re-reads current state sends something different from what was originally intended.

Alerting on Any Arrival

The natural instinct is to alert above a threshold, and it is wrong here. In a healthy pipeline the dead-letter queue is empty, and every entry represents a specific person who did not receive a specific message. A threshold of ten says nine undelivered password resets are acceptable.

Alert on any arrival, and route by reason. A handler_error is an engineering bug and should page. A permanent_4xx for a malformed address is a data-quality issue and can be a ticket. A burst of 404 unknown template immediately after a deploy is a release problem and should page loudly, because it will affect every message using that template until it is rolled back.

Variant: Replaying a Dead Letter

Replay is not simply re-enqueueing. Three checks belong in front of it.

Is the message still useful? A password reset dead-lettered four hours ago references a token that has expired. Replaying it delivers a broken link and a confusing experience; the correct action is to discard it and let the user request another. Check the message's own validity window before replaying anything.

Has the cause actually been fixed? Replaying into the same failure re-dead-letters immediately and doubles the noise. Replay one entry first, confirm it succeeds, then replay the rest.

Will the idempotency key still protect you? If the original send may have succeeded — the ambiguous class above — replaying with the original key is safe and replaying with a new one produces a duplicate. Preserve the key from the stored job rather than generating a fresh one.

Pre-replay checks Before replaying, confirm the message is still useful, that the underlying cause is fixed, and that the original idempotency key is preserved. Three Checks Before Anything Is Replayed 1 — still useful? An expired reset token delivers a broken link. discard rather than replay 2 — cause fixed? Replay one entry, confirm it succeeds, then the rest. never replay in bulk first 3 — key preserved? Reuse the stored key, so a send that already happened is not repeated. Replaying the entire queue after a fix is the single most common way a dead-letter recovery causes duplicate mail.
Replay is a deliberate, staged operation — the queue is a record of individual failures, not a batch to be flushed.

Pipeline Integration

Put the dead-letter store where an operator can query it without shell access to a broker — a database table works well and makes triage a matter of a filtered query rather than a message-by-message inspection.

Emit each dead-letter into the same event store that receives delivery events, so a message's life reads as one timeline from enqueue through attempts to its terminal state. Without that, an operator investigating "did the customer get their receipt" has to correlate two systems by hand.

Set retention long enough to survive a weekend — a failure that starts on Friday evening should still be triageable on Monday — and expire entries after that so the store does not accumulate indefinitely.

Validation and Deployment Checklist

Debugging: Named Symptoms

Symptom: the dead-letter queue is always empty, but users report missing mail. Cause: failures are being swallowed — a bare except that logs and acknowledges, or a worker that crashes without releasing the message. Fix: make the terminal path explicit; every code path out of the handler should either acknowledge a success, requeue, or dead-letter.

Symptom: the same message appears in the dead-letter queue repeatedly. Cause: a replay loop, where a bulk replay re-enqueues everything including entries whose cause is unfixed. Fix: mark entries as replayed and exclude them from subsequent bulk operations.

Symptom: entries carry a reason but no usable payload. Cause: the job was stored by reference and the referenced record has since changed or been deleted. Fix: store the full rendered job, including the recipient and idempotency key, at the moment of failure.

Exit paths from a send handler A handler has exactly three correct outcomes — acknowledge, requeue, or dead-letter — and any other exit silently loses the message. Three Exits, and No Fourth handler returns acknowledge the send succeeded requeue retryable, budget remaining dead-letter permanent or exhausted
A handler that returns without taking one of these three actions has lost the message, and no metric will show it.

Frequently Asked Questions

Should a rate-limit refusal ever dead-letter?

No. A 429 is not a failure — it means the message is early. It should requeue with the provider's stated delay and without consuming a retry attempt. A pipeline that dead-letters on throttling will lose mail during exactly the busy periods when it matters most.

How long should the retry budget be before dead-lettering?

Match it to how long the message stays useful rather than to a generic default. A verification code valid for ten minutes should stop retrying after about six; an invoice notification can retry for hours. A single global budget guarantees one of those two is wrong.

What if the dead-letter queue itself fills up during an outage?

That is the signal that the failure is systemic rather than per-message, and the correct response is different: pause the workers instead of continuing to dead-letter thousands of messages. Add a circuit breaker that trips when the dead-letter arrival rate crosses a threshold, so a provider outage produces a paused queue rather than an emptied one.

Should a dead letter notify the user?

Usually not directly, but the application should know. A receipt that never arrived can be surfaced in the account's message history with a re-send control, which is far better than silence and avoids sending an email about a failed email. What should never happen is the failure being visible only to engineers while the customer is left assuming the message is on its way.

Can we just log failures instead of using a queue?

Logs are not replayable. The point of the dead-letter store is that the original job survives in a form you can act on — a log line records that something failed, while a stored job lets you send it once the cause is fixed. Logs are a useful complement, not a substitute.

← Back to Email Queueing and Retry Architecture