Backpressure Is a Contract Every Caller Must Honor, Not a Setting

On May 29 of this year, Azure OpenAI had the kind of day that belongs in a systems curriculum. Microsoft's public status history describes it plainly: an upstream rollout changed how certain capacity-related failures were surfaced, internal retry traffic grew rapidly in response, and the amplified retries overwhelmed a shared inference load-balancing component. Failures and latency spread across regions and models. Customer impact began at 09:20 UTC. Mitigation was confirmed at 17:05 UTC — almost eight hours later — after Microsoft isolated the offending internal workload onto dedicated infrastructure and rolled back the triggering change.

Strip away the AI context and the shape is old. A component under pressure told its callers it was failing. A caller — Microsoft 365, first-party, running inside the same company — responded to that signal with more traffic. The overload did not stay where the limit was enforced. It relocated upstream to shared infrastructure, and from there it hit bystanders in multiple regions. Microsoft's own account names the gap precisely: that internal workload was not subject to the rate-limiting and overload controls applied to external traffic, and it reached the shared routing layer by a more direct path than external traffic takes. Hence the repair — move the large first-party workloads onto dedicated routing infrastructure, so they stop sharing a load-balancing layer with everyone else.

In codebase after codebase I read, backpressure is a setting. A queue capacity here, a semaphore there, a Reactive Streams dependency in the build file, and the box is checked. The incident reads differently to me, and it matches what I keep re-learning in my own load tests: backpressure is a contract with obligations on both sides of every hop. The service under load owes its callers a slow-down signal that is cheap to produce and early enough to act on. Each caller owes a real reaction — fewer in-flight requests, budgeted retries, or deliberate shedding of its own. A limit enforced in one service and ignored by its callers does not protect the system. It picks the next crash site.

One slow dependency, traced outward

The smallest chain that shows the mechanics has four parts: an edge service A, two internal services B and C, and a database under C. C normally serves 2,000 requests per second. A bad query plan lands, and C's ceiling drops to 400.

What C does next is the first clause of the contract. If C queues incoming work without bound, its latency climbs as the backlog grows. Callers time out long before the backlog drains, so C spends its remaining 400 requests per second answering callers who already hung up, while memory grows until the process dies. The SRE book's overload chapter describes this end state exactly: work piles up, tasks run out of memory and crash, and the failure spreads to neighbors.

So C bounds its queue and rejects the excess quickly. This is the part everyone ships, and the point where most systems stop. C now survives: it serves 400 requests per second of real work and returns fast errors for the rest. But those rejections land on B, and B's reaction — not C's queue — decides whether the incident is over.

B has two paths. If B honors the signal, it caps its in-flight requests toward C, stops retrying into the rejections, fails or degrades the requests it cannot serve, and passes the same signal up to A. Load above C's ceiling gets converted into fast, explicit errors at the edge. Users see a degraded feature. The system stays up.

If B ignores the signal — keeps its concurrency open-loop and retries each failure three times — the offered load on C triples exactly when C is weakest. C keeps rejecting, so B's threads and connection pools fill with attempts-in-progress, B's own latency climbs, and B becomes the overloaded service. The collapse has moved one hop closer to the user. A now faces the same choice B just failed, with less time and more traffic.

The diagram below is the shape I keep sketching in my notes: the slow-down signal leaves the overloaded dependency, crosses the hops that honor it, and dies at the first one that does not — and that hop is where the queue overflow and the collapse reappear.

​

Retries set the speed of that relocation. Google's SRE book works with a per-request budget of three attempts. Compounded across a chain, that number stops being small: if three layers each retry a failing call three times, one user request becomes up to 3³ = 27 attempts against the bottom dependency, and the user's own retry adds another factor of three on top. Azure's post-incident review puts a measured figure next to that arithmetic: in some cases a single failed request produced up to 48 additional attempts, because several layers of the calling stack retried immediately and without enough backoff or jitter. That arithmetic is why the SRE book's rule is strict: a failed request should be retried at exactly one layer — the one immediately above the rejection — and every layer above that should see an error or a degraded response, not a fresh attempt.

What the callee owes

The overloaded service's side of the contract comes down to three obligations, and none of them is "have a queue."

A bound sized in time, not items. A queue bound is a latency promise. The waiting time a queue adds equals its depth divided by the service rate, so a 5,000-item queue in front of a service doing 400 requests per second promises the last caller a 12.5-second wait. If callers give up at 2 seconds, every slot past 800 holds a request that is already dead — the service will process it, and nobody will read the answer. The sizing rule I keep in my notes: bound = service rate × the deadline callers actually use, minus typical processing time. A larger number does not buy resilience; it buys dead work.

Rejection cheaper than service. Saying no must cost close to nothing, because under real overload it becomes the most frequent operation the service performs. The SRE book is blunt about the failure mode: rejecting a request still burns CPU, and a backend can end up overloaded purely by producing rejections. That argues for rejecting as early in the request path as possible — before expensive parsing and lookups — and it sets the quality bar for the whole exercise: a task provisioned for a given rate should keep serving that rate no matter how much excess arrives. Google designs for stability at two to ten times the provisioned load. Goodput — the share of throughput whose results callers actually consume — is the number to defend; total throughput under overload is vanity.

A signal that says what to do next. Not all rejections mean the same thing, and the callee is the only party that knows which one applies. Three messages cover the cases in my notes:

• Retry elsewhere: this replica is busy, but the pool probably is not. The default meaning of a fast, retryable error.
• Slow down, retry later: the pool is saturated. Carry a hint — HTTP's Retry-After header, or gRPC's grpc-retry-pushback-ms metadata, which tells the client exactly how many milliseconds to wait.
• Stop: retrying will not help. gRPC encodes this as a negative pushback value; the SRE book uses an explicit "overloaded; don't retry" error so that deep stacks stop hammering.

Measured against those three, a timeout is the worst signal a callee can emit: it arrives at the last possible moment, costs a full deadline of caller-side resources to receive, and says nothing about what to do next. A service that only signals overload by timing out has, in contract terms, gone silent.

What the caller owes

The caller's side is where the contract usually breaks, because the caller's incentives point the wrong way: from inside one service, retrying harder and queueing more both look like diligence.

A cap on in-flight requests. The minimal honest reaction is a closed loop: a fixed limit on concurrent requests toward each dependency. Little's law does the rest — concurrency equals throughput times latency, so when the dependency slows down, a fixed concurrency cap forces the caller's throughput down automatically. The cap translates the callee's rising latency into the exact response it needs: fewer requests. An open-loop caller that fires a fixed rate regardless of responses has no such coupling; it is the load-test tool that finds the collapse, running in production.

A retry budget, not a retry count. A retry count is a statement about one request. A budget is a statement about the whole client, and only the budget stops amplification. The implementations I compared while writing this converge on the same range:

• The SRE book layers a 10% per-client retry ratio on top of the three-attempt cap; in its worst-case analysis, that takes retry amplification from roughly 3× down to 1.1×.
• gRPC's retry design keeps a token bucket per server name: every failure costs one token, every success refunds tokenRatio (0.1 in the spec's example), and all retries stop while the bucket sits at or below half of maxTokens. The client library also hard-caps maxAttempts at 5, whatever the service config asks for.
• Finagle's default RetryBudget allows about 20% of total requests as retries, plus a floor of 10 retries per second so low-traffic clients are not starved, tracked with a leaky token bucket whose deposited tokens expire on a TTL. The docs carry a warning worth framing: share one budget across every retry layer in the client, or the layers multiply.

The common property is the inversion. Naive retries are most aggressive exactly when the system is sickest. Budgeted retries are plentiful when failures are rare and vanish when failures are common — the retry logic itself honors backpressure.

Context the callee can shed with. A caller that propagates its deadline lets the callee skip work that can no longer be consumed; gRPC applies one call deadline across all attempts of an RPC, so retries cannot resurrect a request whose caller has moved on. Criticality does the same for importance: Google propagates one of four criticality values automatically through RPC metadata, so a task deep in the stack can reject batch work while protecting interactive work without asking anyone. Which requests to shed first is a subject of its own; the contract clause here is only that the label must travel with the call, because the callee cannot invent it.

Self-regulation before the wire. For quota-style rejection there is a stronger caller-side move: stop sending locally. The SRE book's adaptive throttling has each client track its requests and accepts over a two-minute window and reject new calls locally with probability (requests − K·accepts)/(requests + 1). With the default K = 2, the client starts self-limiting once the backend rejects half its traffic; tightening K to 1.1 means the backend only has to reject one request for every ten it accepts. The caveat comes from the same chapter: clients that call rarely hold a stale view of the backend, so this never replaces the callee's own bounds — it moves most of the rejection cost off the wire for the callers that matter.

Granted demand, where the runtime supports it. Everything above is reactive — the caller sends, the callee pushes back. Credit systems invert the default: nothing moves until the receiver grants capacity. Reactive Streams' request(n) is this clause expressed as an interface; HTTP/2 enforces it per stream with flow-control windows that start at 65,535 bytes until the receiver raises them; broker consumers do it with prefetch or permit counts. Inside a process or a single connection, granted demand is the strongest form of the contract because the runtime enforces it. Across service boundaries and fan-outs it decays into the reactive form — which is why the reactive obligations above still matter everywhere.

Where the contract silently breaks

Every backpressure failure I have studied lands in one of four gaps, and none of them looks like a missing setting.

A broker hides the counterparty. Put a queue between two services and the producer's view changes: the broker always accepts, so the producer sees a healthy dependency while the consumer drowns. The slow-down signal still exists — it is called consumer lag — but it lives in a dashboard the producer's code never reads. Fred Hébert's essay Queues Don't Fix Overload called this out back in 2014: with a bigger buffer, "you're making failures more rare, but you're making their magnitude worse." The contract survives a broker only if someone re-establishes it: bound the queue in time (max message age, TTL), or wire lag back to producers as quotas or hard limits. Fire-and-forget is signal amputation — a fine choice when nobody is waiting for the result, and a silent debt when someone is.

Retries invert the signal. A rejection means slow down. A naive retry loop translates it as send again, now. This is the exact clause the Azure incident broke: the triggering change did not remove capacity signals, it changed how they surfaced, and caller-side retry logic turned them into multiplied traffic. Any hop whose retry policy has no budget holds this inversion in reserve, waiting for the right bad day.

Load balancers dilute it. Retrying against a different replica is legitimate — the SRE book relies on it as organic load balancing when a single task runs hot. But when the whole pool is saturated, per-replica rejections retried across the pool convert one node's signal into everyone's load. That is why the stronger designs make the signal collective: gRPC throttles retries per server name rather than per connection, and Google's backends track how many attempts each incoming request has already survived, switching to a stop signal when the histogram says the pool, not the task, is the problem.

Autoscaling races it. Scaling out answers overload with capacity — minutes later, after provisioning, image pull, warm-up, and cache fill. The slow-down signal operates in milliseconds. Without the contract holding the line in between, the retry storm arrives long before the capacity does, and the fleet scales into a system that is already collapsing. Autoscaling is the second line of defense; it cannot be the first.

Blocking or dropping

Hébert frames the end state as a choice that cannot be delegated: when input exceeds capacity for long enough, either block the producers or drop the excess. Both honor the contract; they differ in who absorbs the pain.

Blocking — bounded queues that push wait time upstream, credit systems, in-flight caps — preserves work and fits cooperative callers: internal services, batch pipelines, anything that can genuinely slow down. It has a sharp edge: blocked threads are themselves a resource, and a blocking chain with a cycle in it can deadlock the way any lock cycle does.

Dropping — load shedding, degraded responses — fits the edge, because the public internet does not slow down when asked. Users retry, browsers retry, and mobile SDKs retry on their own schedules; the only defensible move at that boundary is to make rejection nearly free and the degraded path genuinely useful.

Real chains need both: shed at the edge, block between services, and make sure the two meet at a hop that knows which side it is on.

What I take back to my own designs:

• Size every queue in time: depth ÷ service rate must sit inside the deadline callers actually use. Beyond that line, the queue stores dead work, not resilience.
• Make rejection the cheapest operation the service performs, and make it say one of three things: retry elsewhere, retry later (with a hint), or stop.
• Cap in-flight requests toward every dependency. The cap is what turns a slow callee into a slower caller instead of a dead one.
• Replace retry counts with a shared budget in the 10–20% range — the range SRE practice, gRPC, and Finagle all landed on — and retry at exactly one layer.
• Propagate deadlines and criticality with every call; a callee can only shed well what its callers label.
• Treat every broker as a place the signal disappears. Bound it in time, or wire consumer lag back into a producer-visible limit.

Reach for backpressure when callers can cooperate — service-to-service paths, pipelines, anything internal where a concurrency cap and a retry budget are within reach. Prefer shedding where they cannot — the public edge, sporadic clients, cross-organization boundaries. In either case, spend the review time on the callers, not just the service: the queue bound configured this sprint only decides where the next incident lands. Whether it lands at all is decided by every caller above it.