Redundancy & Failover
Everything Adda runs on fails eventually — the question is whether Fahim notices. Mou's answer: a spare that's always ready to take over.
The problem
It's 2am and a single machine — the one running Adda's whole checkout service — has a power supply die. Orders stop. Revenue stops. Mou's phone starts buzzing. The fix is simple: plug in another box. But that takes 40 minutes of someone driving to the data center at 2am, and for those 40 minutes Adda is completely dark — and 40 minutes is nearly Adda's entire monthly error budget spent in one night.
The uncomfortable truth Mou keeps repeating: every component fails eventually. Disks wear out, memory flips bits, a deploy corrupts state, a whole rack loses power. You cannot make any single thing never fail. So the real question isn't "how do I stop failure?" — it's "how do I make sure Fahim doesn't notice when it happens?"
A first attempt
The obvious move is to buy a better machine. Pricier hardware, redundant power supplies inside the box, ECC memory, enterprise disks. Make the one server as bulletproof as money allows. Tanvir has actually floated this.
It helps a little and misses the point entirely. A single server, no matter how premium, is still a single point of failure — one thing whose death takes down all of Adda. You've reduced the odds of that one box dying, but you haven't removed the fact that its failure is your failure. And you still can't survive the switch it's plugged into dying, the rack losing power, or a bad deploy from Shuvo. One of anything is a liability.
The insight
Stop trying to make one thing unbreakable. Instead, Mou's rule: have more than one of everything, so any single failure has a standby to fall over to. That's redundancy. The act of shifting traffic from the dead one to a healthy one is failover.
The design questions become: how many spares, and are they sitting idle or already doing work? That single choice — idle spare vs. working spare — splits every redundancy design into two families.
How it works
Remove the single points of failure
Mou walks Adda's request path and asks "what dies takes us down?" One load balancer, one database primary, one region — each is a SPOF. Duplicate each one. The goal is that no single box, disk, or link is irreplaceable.
Choose active-passive or active-active
Active-passive: one node serves traffic, a standby sits ready. Simpler, but the standby's capacity is paid-for and idle. Active-active: all nodes serve traffic at once and cover each other. Better utilization and no cold spare, but now Adda must handle concurrent writes, split-brain, and load rebalancing.
Detect failure with health checks
The load balancer or orchestrator pings each node every few seconds (GET /healthz). Miss N checks in a row and the node is marked unhealthy and pulled from rotation. The detection window — how fast Adda notices — sets the floor on recovery time.
Fail over automatically
Once a node is marked dead, route around it: the load balancer stops sending it requests, or the standby is promoted to primary. Automate this. A human-in-the-loop failover at 3am costs Mou 20 minutes; an automatic one costs seconds.
Test the failure before it tests you
Untested failover is a coin flip. Mou's team practices killing nodes on purpose — chaos drills — so they learn that the standby's config drifted, or promotion takes 90 seconds, before a real outage teaches them the same thing under pressure.
Active-passive vs. active-active, side by side:
Adda's real numbers
Recovery Time Objective (RTO) — how fast Adda is back — is dominated by detection plus promotion:
- Health check every 2s, mark dead after 3 misses = ~6s to detect. DNS/LB reroute ~5s. Total blip ≈ 10–15s for Adda's stateless active-active web tier.
- Promoting a database standby is slower: detect ~10s, replay the last WAL, repoint the app, warm the cache — often 30–120s, sometimes minutes. Stateful failover is always harder than stateless, as Shuvo learned wiring up replicas back in Part 2.
- Capacity math matters in active-active: if two nodes each run at 60% and one dies, the survivor needs 120% — it falls over too, and now Adda has a cascading outage. Run active-active nodes at ≤ 50% (for 2) so a survivor can absorb the load. This "N+1" headroom is the tax Adda pays for surviving one failure.
When to use it
Active-active can cause split-brain
If two active nodes both think they're the writer — because the network between them broke, not the nodes — they accept conflicting writes and diverge. This is split-brain, and it corrupts Adda's data. Guard against it with a quorum, a consensus protocol (Raft/Paxos), or a fencing token so only one side can win a partition.
Redundancy multiplies availability
Two independent nodes at 99% each give a combined 1 − (0.01 × 0.01) = 99.99% — if their failures are truly independent. That "if" is the catch Mou drills into the team: a shared power supply, shared config push, or shared bug fails both at once. Real redundancy means spreading across failure domains (racks, zones, regions), not just adding boxes.
Practice
Recap
- Every component fails; redundancy means more than one of everything so a spare can take over before Fahim notices.
- Active-passive is simpler but pays for an idle, possibly-rotten standby; active-active uses everything but must handle concurrent writes and split-brain.
- Failover speed (RTO) is detection + promotion; automate it, leave capacity headroom, and rehearse it before a real failure does.
Availability & SLAs
The uptime target that redundancy exists to protect.
Replication
Keeping the standby database's copy of the data current.
Load Balancing
The health checks and routing that make failover automatic.
In an interview
When asked to "make it reliable," don't just say "add a replica." Name the SPOFs one by one, then pick active-passive or active-active per tier and justify it: stateless web tier goes active-active for utilization, the database goes primary-with-replicas because promotion is delicate. Always mention how failover is detected and triggered, and call out split-brain and capacity headroom — those two details separate a real answer from a hand-wave.
How is this guide?
Last updated on
Availability & SLAs
Adda promises its users it will always be there. Mou joins the night uptime becomes a real job — and the first thing she does is turn that promise into a number.
Monitoring & Observability
Adda got slow for hours and nothing paged — because nobody was watching the right thing. Mou can't fix what she can't see, so she gives Adda eyes: logs, metrics, and traces.