Consistency & the CAP Theorem
The network link between Adda's two regions goes dark, and its shards start disagreeing. Ria must choose: stay consistent or stay available. She can't have both — that's CAP.
The problem
Adda now runs its shards and replicas across two datacenters so that if one burns down, the other keeps serving. One afternoon the network link between them goes dark. Both sides are alive and taking traffic — Fahim in Dhaka, his cousin routed to the other region — but they can no longer talk to each other. A user in region A updates their bio; a request in region B reads the old one.
Now Ria faces a decision with no comfortable answer. Does she let both sides keep serving, knowing they may disagree? Or refuse writes on one side to keep everyone in agreement, making that side effectively down? The network split forced a choice she cannot avoid.
A first attempt
The instinct is "just keep both up and reconcile later." Sometimes you can — but reconciliation means two versions of the truth, and for a wallet balance or the last ticket to a concert "later" is too late; you might sell the same seat twice.
The opposite instinct — "block writes until the link heals" — keeps everyone consistent but means an outage every time the network hiccups, which on a system Adda's size is constantly. Neither instinct is free. What Shuvo is bumping into is not a bug you can engineer away; it is a theorem about what is possible when machines cannot communicate.
The insight
The CAP theorem says that when a network partition (P) happens, a distributed system can preserve at most one of Consistency (C — every read sees the latest write) and Availability (A — every request gets a non-error response). Partitions are a fact of networks, so the real question is how you behave during one:
- CP (choose consistency): refuse or block requests on the minority side rather than serve stale or conflicting data. Correct, but partially unavailable during the split.
- AP (choose availability): keep serving on both sides and reconcile afterward, accepting temporary disagreement. Always up, but reads can be stale.
When there is no partition, you get both C and A. CAP is only a dilemma during the split.
How it works
Normal operation: consistency and availability coexist
While the network is healthy, writes propagate and every replica agrees. CAP costs you nothing here — the trade only appears under failure.
A partition splits the cluster
A link dies or a node is isolated. Now some replicas cannot hear about writes happening on the other side. The system must decide, per request, how to respond.
CP systems sacrifice availability
To avoid serving stale data, the minority side rejects reads/writes (or blocks) until it can reach a quorum. Callers see errors or timeouts, but no one ever reads a wrong value.
AP systems sacrifice consistency
Both sides keep answering from their local copy. Writes are accepted everywhere and merged once the link heals — using timestamps, version vectors, or last-write-wins — so reads may be stale until convergence.
After healing, state converges
The partition ends, changes flow both ways, and replicas re-agree. AP systems call the steady state they reach eventual consistency.
Concrete numbers
"Consistency" is really a spectrum, and where you sit on it is a latency knob. Strong consistency across regions means a write waits for a quorum acknowledgment — a cross-continent round trip of 100–300 ms per write. Eventual consistency acknowledges locally in 1–5 ms and converges in the background, usually within tens of milliseconds but seconds under a partition.
That is why Adda makes the choice per operation, not per system. A user's feed or a like-count is happy being AP — a stale count for 200 ms harms no one. A future Adda wallet balance or a limited-drop ticket wants CP — better a brief error than double-charging. Quorum systems tune this directly with R + W > N (read and write replica counts): raise them for consistency, lower them for availability and speed.
When to use it
The trade-off
There is no "CA" system in the real world, because partitions will happen whether you plan for them or not — so you are really choosing CP or AP. Choosing consistency costs availability during splits; choosing availability costs correctness during splits. Decide based on what a wrong answer costs your users.
PACELC: the half of the story CAP leaves out
CAP only describes behavior during a partition. PACELC adds the everyday case: Else (no partition), you still trade Latency vs Consistency. Even on a healthy network, strong consistency costs round trips. So the trade-off is always present — partitions just make it stark.
Practice
Recap
- During a network partition you can keep at most one of consistency and availability — that is CAP.
- CP refuses service to stay correct; AP stays up and reconciles later (eventual consistency).
- The trade is really per-operation and per-latency (PACELC): decide by what a wrong answer costs.
Replication
Sync vs async replication is a CAP choice in disguise.
Transactions
Consistency guarantees within and across machines.
Sharding
Distributing data is what makes CAP bite.
In an interview
Never say "my system is CA." Say "partitions are inevitable, so I choose CP here and AP there," and justify each by the cost of a wrong answer. Bonus points for noting that even without a partition you trade latency for consistency (PACELC). Interviewers want to see you assign consistency per operation, not blanket the whole design.
How is this guide?
Last updated on
Sharding
Adda's data outgrows the biggest machine the cloud rents. Ria has to split it across many — sharding is that split, and choosing the key by user id is everything.
Transactions
On Adda a 'like' must record the like AND fire the notification — a crash halfway can't leave one without the other. Transactions make "all or nothing" real.