Rate Limiting Transactional Sends With a Token Bucket

Every ESP enforces a sends-per-second ceiling, and exceeding it costs you 429 responses, retry load, and eventually a reduced quota. This guide implements a distributed token bucket in Redis that lets a fleet of workers share one provider limit correctly, including the burst behaviour a fixed-window counter cannot express.

Root Cause: A Fixed Window Is Not a Rate Limit

The simplest limiter increments a counter keyed by the current second and refuses work once the count exceeds the ceiling. It is easy to write and wrong in a specific way: it permits twice the intended rate across a window boundary.

With a limit of 100 per second, a worker fleet can send 100 messages at 00:00.999 and another 100 at 00:01.001 — 200 messages in two milliseconds, entirely within the rules as the counter sees them. The provider, which measures over a sliding period, records a burst at double your limit and responds with throttling.

A token bucket has no boundary to exploit. Tokens accumulate continuously at the configured rate, capped at a burst capacity, and each send removes one. The rate is enforced by the refill speed rather than by a window, so there is no instant at which a fresh allowance appears.

The Exact Fix: An Atomic Bucket in Redis

The bucket state is two values — the current token count and the timestamp of the last refill — and both must be read, updated and written without another worker interleaving. A Lua script gives that atomicity in one round trip.

-- token_bucket.lua — atomic take-or-refuse.
-- KEYS[1] bucket key
-- ARGV[1] refill rate (tokens per second)
-- ARGV[2] burst capacity
-- ARGV[3] current time in milliseconds (passed in — never call TIME in a script
--         that may be replicated, and never trust a worker's own clock alone)
-- ARGV[4] tokens requested
local rate     = tonumber(ARGV[1])
local capacity = tonumber(ARGV[2])
local now_ms   = tonumber(ARGV[3])
local want     = tonumber(ARGV[4])

local state  = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(state[1])
local ts     = tonumber(state[2])

if tokens == nil then          -- first use: start full so a cold worker can send
  tokens = capacity
  ts = now_ms
end

-- Refill for the elapsed time, capped at the burst capacity.
local elapsed = math.max(0, now_ms - ts) / 1000.0
tokens = math.min(capacity, tokens + elapsed * rate)

local allowed = 0
local wait_ms = 0
if tokens >= want then
  tokens = tokens - want
  allowed = 1
else
  -- Tell the caller how long to wait rather than making it poll blindly.
  wait_ms = math.ceil(((want - tokens) / rate) * 1000)
end

redis.call('HMSET', KEYS[1], 'tokens', tokens, 'ts', now_ms)
-- Expire well after a full refill so an idle bucket does not linger forever.
redis.call('PEXPIRE', KEYS[1], math.ceil((capacity / rate) * 2000))

return { allowed, wait_ms }

# limiter.py — the calling side.
class TokenBucket:
    def __init__(self, redis_client, key, rate_per_second, burst=None):
        self.redis = redis_client
        self.key = key
        self.rate = rate_per_second
        # Burst defaults to one second of capacity: enough to absorb jitter
        # between workers without ever presenting the provider with a spike.
        self.burst = burst if burst is not None else rate_per_second
        self.script = redis_client.register_script(open("token_bucket.lua").read())

    def take(self, tokens: int = 1) -> tuple[bool, float]:
        """Return (allowed, seconds_to_wait)."""
        now_ms = int(time.time() * 1000)
        allowed, wait_ms = self.script(
            keys=[self.key],
            args=[self.rate, self.burst, now_ms, tokens],
        )
        return bool(allowed), wait_ms / 1000.0

Returning the wait time rather than a bare rejection is what makes the limiter usable. A worker that knows it must wait 40 milliseconds can requeue with exactly that delay instead of retrying immediately and burning CPU, or worse, treating the refusal as a failure and consuming a retry attempt.

Sizing the Burst

Burst capacity is the parameter people get wrong. Set it to zero and the limiter becomes a strict pacer that rejects any momentary clustering, which wastes throughput because real arrivals are never perfectly even. Set it too high and you hand the provider exactly the spike the limiter exists to prevent.

One second of capacity is a good default: enough to smooth the jitter between a handful of workers polling at slightly different moments, small enough that the provider never sees more than a second's worth arrive at once. Increase it only if you have measured genuine idle time caused by rejections, and never beyond what the provider's own documentation describes as an acceptable burst.

Variant: One Bucket Per Provider and Per Message Class

The bucket key should identify the thing being limited, which is the provider account rather than your service. If two applications share an ESP account, they must share a bucket — separate buckets sum to double the rate at the provider.

Where a provider publishes different limits for different endpoints, use a bucket per endpoint. And where you want to guarantee interactive mail is never starved by bulk traffic, run a reserved sub-bucket: give the bulk lane a limiter set below the account rate, leaving the remainder for the interactive lane's own limiter. Both then draw against the account ceiling without either being able to consume all of it.

Pipeline Integration

The limiter belongs in the worker, immediately before the provider call, and a refusal must requeue without consuming a retry attempt. Treating a rate-limit refusal as a failure exhausts the retry budget on a busy afternoon and dead-letters messages that were never actually rejected by anyone.

Set the configured rate below the provider's documented ceiling — typically 10–20% below — so that clock skew between workers and the provider's own measurement window cannot push you over. Expose the current token level as a metric: a bucket sitting at zero for long periods is the clearest possible signal that your queue depth is limited by the provider rather than by your workers.

Validation and Deployment Checklist

• The bucket is implemented as an atomic script, not as separate read and write calls.
• The current time is passed into the script rather than read inside it.
• The bucket key identifies the provider account, shared by every service using it.
• Burst capacity is roughly one second of tokens unless measurement justifies more.
• The limiter returns a wait duration, and callers requeue with that delay.
• A rate-limit refusal does not increment the message's retry attempt count.
• The configured rate sits below the provider's documented ceiling.
• Interactive and bulk lanes draw from sub-buckets that sum to the account limit.
• Token level is exported as a metric and alerted on when persistently empty.

Debugging: Named Symptoms

Symptom: 429s continue despite a limiter that never refuses. Cause: more than one bucket. Two services, or two deployments of the same service, each with their own key, sum to more than the account ceiling. Fix: confirm the bucket key is derived from the provider account and shared, then check for a stale key left behind by a rename.

Symptom: throughput is well below the configured rate. Cause: usually a burst capacity of zero combined with uneven arrivals — workers are refused during clusters and idle between them. Fix: set the burst to one second of tokens and measure again before adjusting the rate itself.

Symptom: the bucket goes negative or refuses forever. Cause: passing a clock that moves backwards, which makes the elapsed time negative and the refill zero. Fix: the script's math.max(0, ...) guard covers this, but the underlying cause is usually a worker with a badly skewed clock — pass a time source you trust.

Symptom: retry attempts exhaust during busy periods with no provider errors. Cause: rate-limit refusals being counted as attempts. Fix: requeue without incrementing; a refusal means the message is early, not failed.

Frequently Asked Questions

Why not just use the provider's 429 responses as the limiter?

Because by then you have already made the request, and providers count rejected attempts against you. Sustained 429s lead to reduced quotas and, at some providers, temporary suspension. Client-side limiting means the provider never sees the excess, and the 429 handling becomes a safety net rather than the primary mechanism.

Does a token bucket work with multiple worker processes?

That is what the Redis implementation is for — the state lives outside the workers, so any number of them share one bucket correctly. An in-process bucket per worker multiplies the effective rate by the worker count, which is the most common way this goes wrong when a service scales horizontally.

What if Redis is unavailable?

Decide the failure mode deliberately. Failing closed stops all sending during a Redis outage, which is safe but turns a cache problem into a delivery outage. Failing open risks a provider throttle. The usual compromise is a conservative in-process fallback — a per-worker rate of the account limit divided by the expected worker count — which keeps mail flowing at reduced throughput without exceeding the ceiling.

Should the limit be per second or per day?

Both, as separate buckets. The per-second rate protects against bursts; a daily quota, which many providers also enforce, needs its own counter with a much longer window. Trying to express a daily quota as a token bucket produces a refill rate so slow that the burst capacity does all the work, which is not a useful limiter.

---

Related

• Email Queueing and Retry Architecture — the worker loop this limiter sits inside
• Designing a Dead-Letter Queue for Failed Sends — where messages go when retries genuinely are exhausted
• ESP Selection and Integration — where each provider's documented limits are compared
• Warming a Dedicated IP — a second, slower limit the same worker must respect

← Back to Email Queueing and Retry Architecture