Design a Distributed Rate Limiter

The problem

Nabila never liked the rate limiter the team bolted on back in Part 3, the night a bot hammered Adda's API. It worked, but it was a quick patch. So the team runs a proper design drill to build it end to end. The scenario is the one that started it: a single abusive client hammers Adda's API 10,000 times a second, and the database — sized for normal traffic — falls over, taking every other user down with it. They want a bouncer at the door: "this key gets 100 requests per minute, no more." On one server that's a counter in memory. Done in five minutes.

Then Tanvir reminds the room that Adda now runs a hundred API servers behind a load balancer, and any request can hit any one of them. Each server sees only its own slice of the traffic. A client doing 100 req/min spread across 100 servers looks like 1 req/min to each — so the "100/min" limit silently becomes "10,000/min". The whole difficulty is that the count must be shared.

Requirements

Functional

• Limit requests per client (by API key, user ID, or IP) to N per time window.
• Reject over-limit requests with HTTP 429 and a Retry-After header.
• Configurable rules per endpoint / per tier.

Non-functional

• Low latency — the limiter is on every request's hot path, so it must add < 5 ms.
• Accurate enough that the limit means something, without a synchronous global lock per request.
• Highly available: if the limiter dies, decide fail-open vs fail-closed deliberately.

A scale estimate

Shuvo sketches the load the limiter has to sit in front of:

• 1 M requests/sec across the fleet, each needing a limit check.
• 100 servers, so ~10 K checks/sec/server locally, but the state is global.
• Millions of distinct clients → millions of counters, each tiny (a count + a timestamp).
• Every request does at least one read-modify-write on shared state. That store is the crux.

The insight

You can't keep the count in each server's memory — it has to live in one shared, fast store that all servers read and increment atomically. Redis is the standard choice: an in-memory store (the same kind Adda leaned on for caching in Part 1) with atomic operations (INCR, EXPIRE) and sub-millisecond latency. The real design work is then choosing an algorithm that trades accuracy against cost, because a naive shared counter has a nasty burst bug.

The algorithms

Key decisions and trade-offs

Bottlenecks and how to scale

• Redis throughput: at 1 M checks/sec, use local batching (lease-and-spend) so most requests never hit Redis. Shard Redis by client key if a single node saturates.
• Latency: keep the check in-process where possible; the network round-trip to a shared store is the dominant cost, so amortize it.
• Atomicity: do the read-modify-write as one atomic op — a Lua script in Redis — so two servers can't both "see 99, write 100."
• Coordinated backpressure: a rate limiter is one layer; pair it with backpressure and circuit breakers — Mou's Part 4 tooling — so the system sheds load gracefully rather than just returning 429s.

Practice

Recap

• The distributed challenge is shared state: the count must be global, or splitting traffic across servers defeats the limit.
• Token bucket balances average rate with burst tolerance using two numbers per client; know fixed/sliding window as alternatives.
• Use an atomic op on a fast shared store, lease tokens locally to cut load, and decide fail-open vs fail-closed on purpose.