Skip to main content

Email Queueing and Retry Architecture for Transactional Systems

Design the send queue behind transactional email: rate limiting, exponential backoff, idempotency keys, dead-letter handling, and priority lanes that keep password resets fast.

The moment your application calls an ESP API inline with a user request, you have coupled a password reset to a third party's availability. This section covers the layer that decouples them: the queue that accepts a send intent, the worker that dispatches it, the retry policy that survives a provider blip, and the dead-letter path that stops a poisoned message from consuming a worker forever.

The requirements are unusual compared with an ordinary job queue. A duplicate HTTP request is invisible to a user; a duplicate receipt is a support ticket. A delayed analytics job is fine; a password reset delayed by ten minutes is a failed login. And unlike most work, sends are rate-limited by an external party whose limits change without warning. Those three properties — visible duplicates, latency sensitivity, and external throttling — shape everything below.

Why the Send Cannot Happen Inline

An inline await esp.send(...) inside a request handler fails in four distinct ways, and only one of them is obvious.

The obvious one is provider downtime: the API is unreachable, the call throws, and the user sees an error for an action that otherwise succeeded. The second is latency coupling — a provider having a slow minute turns your signup endpoint into a slow endpoint, and connection pools fill with requests waiting on someone else's infrastructure. The third is throttling: providers enforce per-second and per-day limits, and an inline call has nowhere to put a message it is not allowed to send yet, so it drops it. The fourth is the subtle one: partial success. The provider accepted the message and your HTTP client timed out waiting for the response, so your code retries and sends it twice.

Inline send versus queued send Inline sending ties the user request to the provider's availability and rate limits, while a queue accepts the intent immediately and absorbs provider failure in a worker. Where the Provider's Problems Land Inline in the request user waits ESP API 429 A rate limit, a slow minute or a timeout becomes a failed signup. There is nowhere to park the message, so it is lost. their availability becomes your availability Through a queue enqueue worker ESP API The request returns as soon as the intent is durable. Throttling becomes queue depth rather than a lost message. their outage becomes your latency
The queue converts every class of provider failure into queue depth, which is a condition you can observe and recover from.

The enqueue itself must be durable before the request returns. Writing the send intent to the same database transaction that created the underlying record — the transactional outbox pattern — removes the last inconsistency: either the user was created and the welcome email was queued, or neither happened.

Core Implementation: A Worker With Rate Limiting and Bounded Retry

The worker is where the interesting constraints live. It must respect a rate limit it did not set, back off when the provider says to, distinguish retryable failures from permanent ones, and never send the same message twice.

# sender_worker.py — dispatch queued sends under an external rate limit with a
# bounded, jittered retry policy.
import random
import time
from dataclasses import dataclass

import redis

r = redis.Redis()

# The provider's documented ceiling, minus headroom for any other process that
# shares the account. Exceeding it costs 429s and, eventually, throttled sending.
RATE_PER_SECOND = 12
MAX_ATTEMPTS = 6

# Failures worth retrying: transport problems and provider-side transients.
RETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}


@dataclass
class SendJob:
    idempotency_key: str      # derived from the business event, stable across retries
    to: str
    template_id: str
    payload: dict
    attempt: int = 0


def acquire_slot(bucket: str = "esp:tokens") -> bool:
    """Sliding-window limiter. Returns False when the current second is full."""
    now = int(time.time())
    key = f"{bucket}:{now}"
    used = r.incr(key)
    if used == 1:
        r.expire(key, 2)          # keys age out on their own; no cleanup job
    return used <= RATE_PER_SECOND


def backoff_seconds(attempt: int, retry_after: float | None) -> float:
    """Honour the provider's Retry-After when present, else exponential + jitter."""
    if retry_after is not None:
        return retry_after        # the provider knows better than our formula
    base = min(2 ** attempt, 300)  # cap at five minutes
    # Full jitter: without it, every worker retries in lockstep and re-storms
    # the provider at exactly the same instant.
    return random.uniform(0, base)


def handle(job: SendJob, esp, queue) -> None:
    if not acquire_slot():
        # Not a failure — we are simply early. Requeue without burning an attempt.
        queue.requeue(job, delay=1.0)
        return

    try:
        # The idempotency key is what makes a redelivered job safe: the provider
        # recognises the repeat and returns the original result.
        esp.send(
            to=job.to,
            template_id=job.template_id,
            payload=job.payload,
            idempotency_key=job.idempotency_key,
        )
    except esp.HttpError as exc:
        if exc.status not in RETRYABLE_STATUS:
            # 400-class errors are deterministic: a malformed address, a template
            # that no longer exists. Retrying cannot change the outcome.
            queue.dead_letter(job, reason=f"permanent {exc.status}: {exc.message}")
            return

        job.attempt += 1
        if job.attempt >= MAX_ATTEMPTS:
            queue.dead_letter(job, reason=f"exhausted after {MAX_ATTEMPTS} attempts")
            return

        queue.requeue(job, delay=backoff_seconds(job.attempt, exc.retry_after))
        return

    queue.ack(job)

Three details carry most of the value. Requeue without incrementing the attempt count when the limiter denies a slot — being early is not a failure, and counting it would exhaust the retry budget on a busy afternoon. Full jitter rather than a fixed exponential delay, because synchronised workers retrying at the same instant reproduce the exact load spike that caused the throttling. And honour Retry-After: when a provider tells you when to come back, that number reflects state you cannot see.

Second Pattern: Priority Lanes

Not all transactional mail is equally urgent. A password reset is worthless if it arrives after the user has given up; a monthly usage summary can wait an hour. A single FIFO queue makes the slow, high-volume traffic delay the fast, low-volume traffic — exactly backwards.

Single queue versus priority lanes In one FIFO queue a bulk batch delays a password reset behind it, while separate lanes with reserved worker capacity keep interactive mail fast. A Reset Should Never Queue Behind a Digest One FIFO queue digest digest digest digest reset The reset waits behind 40,000 digests. Nothing is broken; the queue is simply doing exactly what FIFO means. p99 reset latency: tens of minutes Two lanes, reserved capacity interactive lane — resets, verification codes bulk lane — digests, receipts, summaries Workers reserve a share for the interactive lane, so it drains even under a bulk burst. p99 reset latency: seconds
Lanes are only useful with reserved capacity — strict priority starves the bulk lane, and weighted sharing without a floor starves the interactive one.

Implement lanes with reserved worker capacity rather than strict priority. A pool where, say, 70% of workers poll the interactive lane first and 30% poll bulk first guarantees both drain: the interactive lane cannot be starved by volume, and the bulk lane cannot be starved by a sustained trickle of urgent mail. The rate limiter stays global, because the provider's ceiling applies to the account, not to a lane.

Dead Letters and Poison Messages

A message that fails deterministically will fail on every retry, and without a terminal state it circulates forever, consuming both worker capacity and rate-limit slots that real sends need. The dead-letter queue is what makes failure terminal.

What belongs there: permanent provider rejections (a malformed address, a deleted template), messages that exhausted the retry budget, and anything that threw an unhandled exception in the worker itself. What does not: throttling responses, which are not failures at all, and any 4xx that a documented Retry-After accompanies.

The dead-letter queue is only valuable if someone looks at it. Alert on non-zero depth rather than on a threshold, because a healthy pipeline produces zero dead letters and any entry represents a real message a real user never received. Store the full job payload alongside the failure reason so a fixed message can be replayed rather than reconstructed.

Failure Class Worker action Alert
429 with Retry-After Throttling Requeue at the stated delay, attempt count unchanged No
503 from the provider Transient Exponential backoff with jitter Only if sustained
Connection timeout Ambiguous Retry with the same idempotency key Only if sustained
400 malformed address Permanent Dead-letter immediately Yes
404 unknown template Permanent Dead-letter immediately Yes, page — a deploy broke
Retry budget exhausted Terminal Dead-letter with the full attempt history Yes

Pipeline Integration Steps

  1. Write the send intent inside the originating transaction, so a queued message and the record that justifies it commit together.
  2. Derive the idempotency key from the business event — the order ID, the reset token — never from a random value generated per attempt, or every retry looks like new work to the provider.
  3. Split lanes by latency requirement, not by message type: anything a human is actively waiting for belongs in the interactive lane.
  4. Set the limiter below the provider's documented ceiling to leave headroom for other processes sharing the account.
  5. Cap the retry budget at a duration matched to the message's usefulness — a reset token that expires in fifteen minutes should not be retried for an hour.
  6. Dead-letter with the full payload and alert on any non-zero depth.
  7. Emit the outcome into the same event store that consumes webhook delivery events, so a message's life is queryable from enqueue to bounce in one place.

Debugging: Named Symptoms, Cause, and Fix

Symptom: queue depth grows steadily during business hours and drains overnight. Cause: the limiter is set below your actual arrival rate, so you are shipping less mail per second than you generate. Fix: compare peak arrival rate against RATE_PER_SECOND; if the provider's ceiling is genuinely the constraint, request a limit increase rather than adding workers, which will only produce more 429s.

Symptom: a burst of 429s every time the queue recovers from a pause. Cause: retries synchronised by a fixed backoff, all firing at the same instant. Fix: apply full jitter, as in the worker above — the randomised delay spreads the recovery over the whole window.

Symptom: customers report receiving the same receipt twice, minutes apart. Cause: a timeout after the provider accepted the message, followed by a retry with a fresh idempotency key. Fix: derive the key from the business event and verify it is stable across attempts; the ESP integration layer is where that guarantee belongs.

Symptom: the dead-letter queue fills with 404 template errors after a deploy. Cause: a template was renamed or removed while messages referencing it were still queued. Fix: treat template IDs as an API contract — deprecate rather than delete, and drain the queue before removing an ID from the provider.

Symptom: password resets arrive ten minutes late, but only on some days. Cause: a bulk job sharing the queue. Fix: the priority lanes above, with reserved capacity rather than strict weighting.

Validation and Deployment Checklist

Observability: The Four Numbers That Describe a Send Pipeline

A queue that is working and a queue that is quietly failing look identical from the application's side — both accept messages without error. Four metrics distinguish them, and a pipeline that publishes all four can be diagnosed without reading a single log line.

Queue depth is the obvious one, and the least informative on its own. Depth rising is normal during a burst and alarming only if it does not fall; what matters is the trend across a window long enough to cover your traffic shape. Alert on depth that has not decreased over several consecutive intervals rather than on any absolute value, because the threshold that is healthy at midday is a serious incident at three in the morning.

Oldest message age is the number that actually maps to user experience. A queue holding fifty thousand digests with an oldest age of forty seconds is healthy; one holding six messages with an oldest age of twenty minutes has a stuck consumer and a user waiting for a password reset. Publish it per lane, because a single global figure hides exactly the case the lanes exist to prevent.

Attempt distribution — how many messages are on their first attempt versus their third or fifth — reveals provider trouble before it becomes visible anywhere else. A healthy pipeline is overwhelmingly first-attempt. A rising tail means the provider is rejecting or throttling, and it rises well before the dead-letter queue starts filling.

Dead-letter arrival rate is the terminal signal, and it should normally be zero. Because every entry represents a message a real user never received, alert on any arrival rather than on a rate — a threshold implies an acceptable number of undelivered password resets, which there is not.

Four metrics that describe a send pipeline Queue depth, oldest message age, attempt distribution and dead-letter arrivals, each mapped to the condition it detects and the alerting rule that suits it. What Each Number Actually Detects queue depth Detects: throughput below arrival rate. Alert on: no decrease over several intervals. oldest message age Detects: a stuck or starved consumer. Alert on: per-lane latency budget breached. attempt distribution Detects: provider throttling or instability. Alert on: a growing tail beyond attempt two. dead-letter arrivals Detects: messages nobody will ever receive. Alert on: any arrival at all.
Depth is the metric everyone graphs and the least diagnostic; oldest message age is the one that corresponds to what a user experiences.

The pairing that catches the most incidents is depth against oldest age. Rising depth with a stable oldest age is a burst being absorbed correctly — the queue is doing its job. Stable depth with a rising oldest age is far more serious: messages are being consumed, but something in the ordering or the lane assignment is leaving a subset behind, which is the signature of a stuck partition or a lane with no workers assigned. Neither condition is visible from depth alone, which is why a dashboard showing only queue length routinely reports green through an incident.

Frequently Asked Questions

Should I use my ESP's built-in queue instead of my own?

Providers do queue and retry internally once they have accepted a message, and that covers the SMTP-level retries you should never implement yourself. What their queue cannot do is absorb the failure of the API call itself — if the provider is unreachable or throttling you, the message never reaches their queue. Your queue exists to cover the gap between your application and their front door; theirs covers everything after.

How long should the retry budget be for a password reset?

Shorter than the token's lifetime. If the reset link expires in fifteen minutes, retrying for an hour delivers a message that cannot be used and looks broken to the recipient. Set the budget to roughly two-thirds of the token lifetime and dead-letter after that, so the user requests a fresh reset rather than receiving a stale one.

Is a database table an acceptable queue?

For low volume, yes, and it has a real advantage: the enqueue can share a transaction with your application write, which removes an entire class of consistency bug. The limits show up under concurrency — polling workers contend on the same rows, and SELECT ... FOR UPDATE SKIP LOCKED becomes essential. Move to a dedicated broker when contention rather than throughput becomes the bottleneck.

Do I need exactly-once delivery?

No, and you cannot have it. Every practical queue offers at-least-once, so the achievable goal is at-least-once delivery with idempotent effects. The idempotency key on the send call is what converts a duplicate delivery into a duplicate that produces no second email, which is the property that actually matters to the recipient.

Should retries happen in the queue or inside the worker?

In the queue. An in-process retry loop holds a worker for the duration of the backoff, which wastes capacity and loses all state if the process restarts. Requeueing with a delay makes the retry durable, frees the worker immediately, and gives you queue depth as an observable signal of how much retrying is happening.

How do I stop a poison message from taking down the pool?

Bound both dimensions: cap attempts per message so it eventually dead-letters, and cap the worker's per-job execution time so a hang cannot hold a slot indefinitely. Without the second cap, a single message that never returns removes one worker permanently, and a handful of them silently reduce the pool to nothing.

← Back to Transactional Email Delivery Infrastructure