Your Queue Won't Drain: The Arithmetic of Backlog Recovery

The page fires at 3 AM: queue depth 3 million and climbing. I open the consumer dashboard expecting carnage. Instead every consumer is green, CPU is flat, error rate is near zero, and the lag line is dead level. Nothing is broken. Nothing is recovering either. That contradiction is the whole problem, and it is not a mystery — it is arithmetic I failed to do in advance.

I spent a few evenings turning that panic into formulas after reading an InfoQ piece from May 2026 that laid the math out cleanly, then chasing the dangerous part down into the metastable-failures literature. These are my notes: the three numbers that decide whether a backlog ever drains, the cliff that hides the risk, the retry loop that can defeat a correctly sized fleet, and a small TypeScript model that let me watch all three happen. The math is broker-agnostic; I use AWS SQS for the concrete framing because its visibility-timeout redelivery makes the retry loop easy to see.

Three numbers, and the only one that drains a backlog

A queue has three inputs you already watch, even if you never named them. Arrival rate (λ): messages entering per second. Per-consumer processing rate (μ): messages one consumer clears per second. Consumer count (c). Total capacity is c × μ. If capacity beats arrival, the queue stays small. If not, it grows. Everything downstream is a consequence of that one comparison.

The number that matters during recovery is the gap between them:

Surplus is the only capacity available to chew through a backlog, because steady-state arrivals are already consuming the rest. This is where the 3 AM surprise lives. A fleet provisioned to handle steady-state traffic — the rational, cost-conscious default — has a surplus of exactly zero. Its drain time for any backlog is infinite. The lag line is flat not because the system is stuck on a bug, but because backlog ÷ 0 has no finite answer.

Concretely: 10,000 msg/sec arriving, 400 msg/sec per consumer, 25 consumers. Capacity is 10,000. Surplus is 0. A 3.6M-message backlog from a 10-minute incident never clears. Add seven consumers for 32 total and surplus jumps to 2,800/sec; the same backlog drains in about 21 minutes. Those seven instances are the entire difference between "recovers before breakfast" and "never recovers without a human."

That gives the headroom formula directly. To clear a worst-case backlog within a recovery time objective (RTO, in seconds) on top of steady-state demand:

For a 5M worst-case backlog and a 30-minute RTO: 10000/400 + 5000000/(400 × 1800) = 25 + 7 = 32. A 28% overhead above steady-state. The value of the formula is not the number — it is that capacity planning stops being a feelings-based negotiation ("some extra headroom") and becomes a line item you can defend or cut.

The cliff that hides the risk

The reason backlogs feel like they appear from nowhere is that the relationship between utilization and queue growth is non-linear. Utilization is λ ÷ (c × μ). Watch what a single 10% traffic bump does at two starting points.

At 80% utilization on a 10,000/sec fleet, surplus is 2,000/sec. A 10% spike lifts you to 88%; surplus falls to 1,200. Annoying, survivable. At 90% utilization, surplus is 1,000/sec. The same 10% spike pushes you to 99%; surplus collapses to 100/sec. The queue now grows roughly ten times faster than it did at 80% for an identical spike. The system did not change overnight. The margin was just thinner than the dashboard implied, and thin margin is invisible until something leans on it.

Little's Law gives you the customer-impact translation for free, and it holds for SQS, Kafka, RabbitMQ, or a Redis list alike:

If 600,000 messages sit in front of an arrival rate of 5,000/sec, the message arriving now waits about 120 seconds before processing even starts. Flip it: if your SLA is 10 seconds at 5,000/sec, your maximum tolerable depth is 50,000. That second number belongs on a dashboard with an alarm wired to it, because it is the line where you are breaching by definition rather than by judgment.

The retry loop that defeats a correct fleet

Here is the part that turns a backlog into an outage, and the part I most want my future self to remember. Sizing surplus correctly is necessary but not sufficient, because arrival rate is not a constant. It responds to the queue.

When the queue is deep, messages wait longer. Producers waiting on responses time out and retry. With SQS the same dynamic appears without any client code: a message whose visibility timeout expires before the consumer acks is redelivered, so a slow consumer manufactures its own extra arrivals. Each retry is another message. Effective arrival climbs as a function of how backed up you already are:

Timeout probability rises with depth, which raises effective arrival, which raises depth. That feedback loop is a sustaining effect, and it can hold effective arrival above capacity even after the original trigger is gone. The Bronson et al. HotOS '21 paper names this a metastable failure: a stable-but-useless state that persists after its trigger is removed, sustained by an optimization meant for the common case. Marc Brooker's gloss on the queue version stuck with me — the system is "Up, but down." Throughput looks great; goodput is zero, because the callers stopped waiting for the answers.

The numbers say this is not a corner case. In the empirical study of metastable failures across eleven organizations, retry amplification was the sustaining effect in more than half of incidents, and the documented outages ran from 1.5 to 73 hours. The InfoQ author's own example is mundane and exactly on point: an SQS order pipeline lost a downstream payment service for eight minutes, producers retried the whole time, effective arrival came back at 2.5x base, and the queue kept growing for another 40 minutes with every consumer healthy. An 8-minute trigger became an hour of customer-facing pain, and the second 52 minutes were caused entirely by the recovery dynamics, not the original fault.

I wanted to feel the difference rather than trust the formula, so I built a one-file simulator that steps a backlog forward one second at a time. Deeper queue, higher timeout probability, more re-enqueued work.

// queue-drain.ts — drain-time math + a small SQS-style consumer simulation.
// Run: npx tsx queue-drain.ts

type Fleet = {
  arrivalRate: number; // base messages/sec entering the queue (lambda)
  perConsumer: number; // messages/sec one consumer drains (mu)
  consumers: number;   // fleet size (c)
};

// Surplus is the ONLY capacity available to drain a backlog.
export const surplus = (f: Fleet): number =>
  f.consumers * f.perConsumer - f.arrivalRate;

// Static drain time (seconds), ignoring retries.
// Infinity when surplus <= 0: the backlog never clears on its own.
export const drainSeconds = (f: Fleet, backlog: number): number => {
  const s = surplus(f);
  return s <= 0 ? Infinity : backlog / s;
};

// Consumers needed to clear maxBacklog within rto seconds, plus steady-state.
export const headroom = (
  arrivalRate: number, perConsumer: number, maxBacklog: number, rto: number,
): number => Math.ceil(arrivalRate / perConsumer + maxBacklog / (perConsumer * rto));

// Step the backlog one second at a time. A deeper queue raises timeout
// probability, so timed-out producers re-enqueue and arrivals climb.
export function simulate(
  f: Fleet, backlog: number, retriesPerTimeout: number, maxSteps = 36_000,
): number[] {
  const capacity = f.consumers * f.perConsumer;
  const depths: number[] = [];
  let depth = backlog;
  for (let t = 0; t < maxSteps; t++) {
    depths.push(depth);
    if (depth <= 0) break;
    const timeoutProb = Math.min(1, depth / (f.arrivalRate * 30));
    const effectiveArrival = f.arrivalRate * (1 + retriesPerTimeout * timeoutProb);
    depth = Math.max(0, depth + effectiveArrival - capacity);
  }
  return depths;
}

function report(label: string, f: Fleet, backlog: number, retries: number): void {
  const depths = simulate(f, backlog, retries);
  const drained = depths[depths.length - 1] <= 0;
  const mins = drained ? (depths.length / 60).toFixed(1) + " min" : "never";
  console.log(`${label}: surplus=${surplus(f)}/s drain=${mins} peak=${Math.max(...depths).toLocaleString()}`);
}

const steady: Fleet = { arrivalRate: 10_000, perConsumer: 400, consumers: 25 };
const withHeadroom: Fleet = { ...steady, consumers: 32 };
const backlog = 3_600_000;

console.log(`headroom (5M backlog, 30-min RTO): ${headroom(10_000, 400, 5_000_000, 1800)} consumers`);
report("steady-state fleet, no retries", steady, backlog, 0);
report("with headroom,  no retries    ", withHeadroom, backlog, 0);
report("with headroom,  retry storm   ", withHeadroom, backlog, 1.5);

Run it with npx tsx queue-drain.ts. The output is the argument:

The non-obvious line is the third. The 32-consumer fleet that clears the backlog in 21 minutes with no retries never recovers once retriesPerTimeout is 1.5 — the peak runs off to hundreds of millions. The only thing that changed is that arrivals now respond to depth. The single load-bearing line in simulate is timeoutProb = Math.min(1, depth / (f.arrivalRate * 30)): it ties arrival rate to how deep the queue is, which is what converts a draining system into a self-sustaining one. Provisioning did not fail. The feedback loop did.

​

The diagram above traces that loop; the edge to watch is the one that closes it — arrivals climbing because depth climbed.

Shed or drain, and trigger on the slope

Once you accept that arrivals respond to depth, two operational rules follow.

The first is that draining is not always the right move. If estimated drain time exceeds the message TTL, most of the backlog is already garbage — the callers timed out and left. Processing it burns compute on work that helps no one while fresh requests wait behind it. The rule is blunt: if drain_time > message_ttl, shed. Drop messages past their TTL, deprioritize batch traffic behind real-time, and serve degraded responses where a fallback exists. Shedding has a quieter payoff for planning too: if stale work is discarded, worst-case backlog is bounded by the TTL window instead of by incident duration, which shrinks the headroom you must reserve. For large systems, smart admission control is often cheaper than standing idle consumers. This is the backlog-side complement to request-admission load shedding; the levers differ but the goal — drop the cheapest work first — is the same.

The second rule is about when to scale. Triggering on absolute queue depth is too late by construction: by the time depth alarms, you are already deep in the cliff. Trigger on the rate of change instead — rate(queue_depth[5m]) in Prometheus, metric math in CloudWatch — and project forward by your provisioning lag, because a new consumer that needs three minutes to pull an image and start should be sized for where the backlog will be when it arrives, not where it is now. A growing slope under healthy consumers is also your earliest metastable signal, well before depth becomes frightening.

There is a real cost to overcorrecting here, and it is worth naming. Modal behavior — flipping policy under overload — is hard to reason about and easy to get wrong; a shed path that rarely runs is a shed path that rots. Priority schemes assume your workload has a clean priority order, which many do not. And in a multi-stage pipeline, scaling on a local depth signal can send you scaling the wrong stage entirely: throughput is capped by the slowest stage, so adding consumers upstream of the real bottleneck buys exactly zero recovery and a larger bill. Monitor depth at every stage, and fix the bottleneck before you scale anything in front of it.

When to reach for this, and when not

Do the drain-time arithmetic whenever a backlog can form faster than you can add capacity — queue-backed pipelines, async workers, anything with an RTO. Skip the heavy machinery when the queue is shallow by design, when arrivals genuinely do not respond to latency (no producer retries, no redelivery, fire-and-forget), or when the cheapest correct answer is simply more standing surplus.

What I changed after this study, concretely:

• Size fleets with consumers = λ/μ + max_backlog/(μ × RTO), not steady-state. Zero surplus means infinite drain.
• Put the Little's Law SLA depth (λ × SLA_seconds) on a dashboard with an alarm. It is the line you breach by definition.
• Auto-scale on the slope of depth, projected by provisioning lag — never on absolute depth.
• Watch effective vs. base arrival during recovery. Flat-or-growing depth under healthy consumers means amplification; add backoff, jitter, and a DLQ, not consumers.
• Set if drain_time > TTL, shed. Bound worst-case backlog by TTL instead of incident length.

Reach for the math at 3 AM and you divide two numbers and know where you stand. Skip it, and you refresh a flat lag line wondering why a healthy fleet won't move — which is the metastable trap wearing a calm face.

Further reading: the Bronson et al. HotOS '21 paper on metastable failures for the underlying model, and Marc Brooker's writing on metastability for the control-systems intuition.