Backpressure & Circuit Breakers
One slow Adda service starts dragging down everything that calls it. Mou needs a way to fail fast and stop the slowdown from swallowing the whole app.
The problem
Adda's payments service gets a little slow — its database is under load, so calls that took 20ms now take 3 seconds. Not down, just slow. The Orders service calls Payments on every checkout, so its threads start piling up waiting. Within a minute every Orders thread is blocked on a slow Payments call. Now Orders is unresponsive. The web tier that calls Orders backs up too. Mou watches, on the dashboards from last lesson, a tiny hiccup in one service take down the entire app — the traces light up red from the bottom of the stack all the way to Fahim's home feed.
This is the nightmare of distributed systems: failure doesn't stay put. A single slow dependency spreads upward through everything that waits on it, until all of Adda is jammed. And "slow" is worse than "down" — a fast rejection frees the caller, but a slow success holds every caller hostage.
A first attempt
The instinct — Tanvir's first patch — is to add retries and timeouts. If a call is slow, time out after a few seconds and try again. Surely retrying a flaky call helps?
It makes things dramatically worse. When Payments is already overloaded, every timeout spawns a retry, so the struggling service now gets 2–3x the traffic — a retry storm that guarantees it never recovers. And a naive 30-second timeout still holds the caller's thread for 30 seconds; multiply by thousands of Adda's requests and you exhaust the thread pool anyway. Retries and timeouts are necessary, but applied naively they pour fuel on the fire instead of putting it out.
The insight
Mou brings two complementary ideas that break the cascade:
- Fail fast, don't wait. If a dependency is clearly sick, stop calling it immediately and return an error (or a fallback) in microseconds. Freeing the caller instantly is what stops the pile-up. This is the circuit breaker.
- Push back when overwhelmed. If work arrives faster than a service can handle it, signal upstream to slow down or shed the excess, instead of accepting infinite work and collapsing. This is backpressure.
One protects callers from a sick callee. The other protects a callee from too many callers. Together they keep Adda's local failures local.
How it works
Wrap risky calls in a circuit breaker
Put a breaker around every call to a dependency that can fail. Like an electrical breaker in the apartment where Adda started, it has three states — closed (calls flow), open (calls are rejected instantly), and half-open (a trickle of test calls) — and it flips based on the recent error/latency rate.
Trip the breaker on failure
Track a rolling window of results. If failures exceed a threshold — say 50% of the last 20 calls — trip to open. Now every call returns instantly with an error or fallback, without touching the sick service. This is what frees the caller's threads and lets the downstream service breathe.
Probe with half-open, then recover
After a cooldown (say 10s), move to half-open and allow a few probe calls through. If they succeed, close the breaker and resume normal traffic. If they fail, snap back to open and wait again. This auto-recovers without waking Mou up.
Apply backpressure with bounded queues
Give each Adda service a bounded work queue. When it fills, reject new work immediately (return 429) rather than buffering forever. A full queue is the signal "I'm at capacity" — propagate it upstream so callers slow down instead of piling on.
Shed load and degrade gracefully
When Adda must drop work, drop the least important first: reject low-priority requests, serve a cached or default response, disable the recommendations widget but keep checkout alive. A degraded-but-up Adda beats a perfect-but-down one.
The circuit breaker state machine:
Adda's real numbers
- Timeouts must be tight. If Payments' normal p99 is 50ms, Mou sets the timeout at ~250ms, not 30s. A tight timeout frees a thread in a quarter-second instead of holding it half a minute — the difference between absorbing a blip and exhausting a 200-thread pool.
- Thread-pool math. 200 threads and a 3s hung dependency means Orders can serve at most ~66 req/s before every thread is blocked. Drop the timeout to 250ms and the same pool sustains ~800 req/s of failing calls without locking up.
- Retry with backoff and jitter, capped. Retry at most 2–3 times, with exponential backoff (100ms, 200ms, 400ms) plus random jitter, and only when the breaker is closed. Jitter prevents thousands of Adda clients retrying in a synchronized wave.
- Bounded queue size. A queue of 1,000 at 500 req/s adds up to 2s of latency when full — beyond that, new requests should be rejected (429), not queued, or you just build a latency time bomb.
When to use it
An open breaker is still an outage — decide the fallback
Tripping the breaker stops the cascade, but callers now get errors. Mou must decide what they get: a cached value, a sensible default, a queued write to process later, or a clean error. "Fail fast" without a fallback plan just moves the failure — the win is that it's now contained and fast instead of spreading and slow.
Backpressure needs a bounded buffer to work
Backpressure only exists if something can say "full." An unbounded queue can never push back — it silently absorbs load until memory runs out and the whole process dies (often taking the queued work with it). Bounding every queue, thread pool, and connection pool is what converts "infinite silent buffering then crash" into "explicit fast rejection." Limits are a feature.
Practice
Recap
- Failure spreads upward: a slow dependency blocks its callers' threads until all of Adda jams — and "slow" is worse than "down."
- Circuit breakers protect callers by failing fast (closed → open → half-open) when a dependency is sick; backpressure protects a service by pushing back when it's overwhelmed.
- Tight timeouts, capped retries with jittered backoff, bounded queues, and graceful degradation are the concrete tools that keep local failures local.
Monitoring & Observability
The saturation and error signals that trip breakers and trigger shedding.
Message Queues
Bounded queues as a buffer and backpressure mechanism.
Rate Limiting
Capping load at the edge before it ever reaches a fragile service.
In an interview
When a design has a synchronous call to a fragile dependency, proactively say: "I'd wrap this in a circuit breaker with a tight timeout and a fallback, so a slow Payments service can't cascade into Orders." Then describe the fallback explicitly — cached value, queued write, or graceful degradation. Naming the retry storm and the thread-pool exhaustion failure mode shows you understand why the pattern exists, not just that it's a buzzword.
How is this guide?
Last updated on
Monitoring & Observability
Adda got slow for hours and nothing paged — because nobody was watching the right thing. Mou can't fix what she can't see, so she gives Adda eyes: logs, metrics, and traces.
Disaster Recovery
An entire Adda region goes dark and a bad migration nearly wipes real data. Two questions land on Mou at once: how long until we're back, and how much did we lose? DR is the plan she hopes never to run.