Dead-Letter Queues & Retry/Backoff Strategies

Every message queue eventually delivers a message that its consumer can't process. Maybe the downstream database is briefly unreachable, maybe the payload is malformed, maybe a dependency is mid-deploy and returning 500s. The instinct is to retry — and retrying is the right instinct. But the naive version of it, "catch the error and try again immediately, forever," is one of the most reliable ways to turn a small problem into an outage.

Two ways naive retries make things worse

The first failure mode is the poison message. If a message is malformed in a way that will never succeed — a missing required field, a schema mismatch, a bug in the producer — then retrying it immediately and unconditionally doesn't give it a second chance. It just loops forever, burning consumer capacity on work that can never complete, and in many queue implementations blocking every message behind it from being processed at all.

The second failure mode is the retry storm. Say a downstream service degrades under load and starts timing out. Every client that was calling it retries immediately, in a tight loop, which adds even more load to a service that was already struggling. The retries themselves become the reason the service can't recover — a classic thundering-herd problem, and one that gets worse the more clients you have, not better.

Exponential backoff with jitter

The fix for both failure modes starts with spacing retries out. Exponential backoff doubles the delay after each failed attempt, so a struggling dependency gets progressively more room to recover instead of being hit at a constant rate. But backoff alone doesn't solve the retry storm — if every client backs off on the exact same schedule, they stay synchronized, and every retry wave still arrives as a spike. Jitter — adding randomness to the delay — breaks that synchronization by spreading retries out in time.

function nextDelay(attempt: number): number {
  const base = 500; // ms
  const cap = 30_000; // ms
  const exp = Math.min(cap, base * 2 ** attempt);

  // Full jitter: pick anywhere between 0 and the capped
  // exponential delay, so retries from many clients spread
  // out instead of firing in lockstep.
  return Math.random() * exp;
}

// attempt 0 -> up to  500ms
// attempt 1 -> up to 1000ms
// attempt 2 -> up to 2000ms
// attempt 3 -> up to 4000ms
// ...
// attempt 6+ -> up to 30000ms (capped)

This is "full jitter": instead of a fixed exponential delay, each client picks a random value between zero and the capped exponential delay. The cap matters too — without one, a long enough failure streak produces delays measured in hours, which is rarely what you want; capping it means the system settles into a steady retry rate instead of drifting further and further out.

The dead-letter queue as a safety valve

Backoff buys time, but it doesn't solve the poison-message problem — some messages will never succeed no matter how long you wait between attempts. That's what a dead-letter queue (DLQ) is for. After a message fails processing N times, the consumer stops retrying it on the main queue and moves it to a separate DLQ instead.

This does two things at once. It protects the main pipeline's throughput, because one bad message no longer occupies a consumer slot (or blocks head-of-line delivery) indefinitely. And it preserves the failed message rather than discarding it, so someone can actually look at why it failed instead of having the evidence vanish along with the error.

Choosing sensible limits

The retry limit before dead-lettering is a judgment call that depends on how transient the expected failures are. A message failing because of a network blip might be worth 5-8 attempts spread over a few minutes; a message failing because of a validation error will fail identically on attempt one and attempt eight, so it's worth detecting that class of error and dead-lettering it immediately rather than burning through the full retry budget. Not every failure deserves the same number of chances, and treating them all the same is how you end up either dead-lettering a message that would have succeeded on retry three, or wasting ten minutes retrying something that was never going to work.

Reprocessing: closing the loop

A DLQ is only half the story — messages that land there still need to get processed once the root cause is fixed. The common pattern is a replay tool that reads messages back off the DLQ and re-publishes them to the main queue, so they flow through the exact same processing path as any other message rather than through a bespoke, less-tested code path.

This only works safely if processing is idempotent. A message might have partially succeeded before it was dead-lettered — written to one table but not another, say — and replaying it should produce the same end state whether it runs once or three times. Idempotency keys, upserts instead of inserts, and checking for existing side effects before applying new ones are what make replay safe rather than a second source of duplicate writes.