Disaster Recovery
An entire Adda region goes dark and a bad migration nearly wipes real data. Two questions land on Mou at once: how long until we're back, and how much did we lose? DR is the plan she hopes never to run.
The problem
A fiber cut, a flood, a botched config push, or a region-wide cloud outage takes an entire Adda data center offline. Not one server — everything, all at once. The redundant nodes Mou set up two lessons ago were all in that building. And on the same bad night, a migration Shuvo ran deletes a chunk of the users table. Two questions land on Mou immediately, and Ria wants numbers: how long until Adda's back? and how much data did we just lose?
This is a different problem from a single server dying. Redundancy and failover handle component failure inside a healthy site. Disaster recovery is about surviving the loss of the whole site — the rare, correlated catastrophe that takes out everything Adda thought was independent — and the logical disaster where the data itself goes wrong. Mou hopes to never run the plan. She must still have one.
A first attempt
"We take nightly backups" is the classic answer — it's what Adda had — and it hides two fatal gaps.
First, a nightly backup means that when the disaster hits at 4pm, Adda loses up to 16 hours of data — every post, message, and sign-up since midnight is gone. Second — and this is the one that ends careers — an untested backup is not a backup, it's a hope. Teams discover during the real disaster that the backup was silently failing for months, or that restoring 2TB takes 11 hours, or that nobody knows the runbook. A backup you've never restored has an unknown, and usually terrible, recovery time.
The insight
Mou says you can't even talk about disaster recovery without two numbers that turn "how bad?" into a target you can design and budget against:
- RPO — Recovery Point Objective: how much data you can afford to lose, measured in time. RPO of 5 minutes means backups/replication must be at most 5 minutes behind. It sets your backup frequency.
- RTO — Recovery Time Objective: how long you can be down before you're back. RTO of 1 hour means recovery must complete within an hour. It sets your recovery strategy.
Every DR design is just "how small do RPO and RTO need to be, and what is Adda willing to pay for that?" Smaller numbers cost exponentially more — so Mou picks per system, not one target for everything.
How it works
Set RPO and RTO per system
Adda's ledger of payment transactions might need RPO ≈ 0 and RTO of minutes. The analytics warehouse Shuvo runs might tolerate RPO of a day and RTO of a week. Don't buy the strictest tier for data that doesn't need it — DR spend should track business impact.
Pick a DR strategy to match
Four tiers, cheapest to costliest: Backup & Restore (restore from cold storage — cheap, RTO hours), Pilot Light (core data replicated, minimal infra always on, scale up on disaster), Warm Standby (a scaled-down full copy always running, scale up on failover), Hot Standby / Multi-Region Active-Active (full capacity in another region, failover in seconds). Cost climbs with each tier; RTO/RPO shrink.
Replicate across failure domains
Backups and standbys must live in a different region, not the same building or even the same city as Adda's primary. Replicate continuously to hit a small RPO. Store backups following 3-2-1: 3 copies, on 2 media types, with 1 off-site.
Automate the failover and the DNS cutover
When the primary region is declared dead, promote the standby and repoint traffic — usually by updating DNS or a global load balancer to the standby region. Automate it; a manual region failover under pressure at 4am is where RTOs quietly triple.
Run the fire drill regularly
Mou's team actually fails over to the DR region on a schedule (game days). This is the only way to verify Adda's real RTO/RPO, catch config drift, and make sure everyone knows the runbook. An untested DR plan has an effective RTO of infinity.
The DR strategy spectrum — cost vs. recovery:
cheaper, slower ◄─────────────────────────────► costlier, faster
┌──────────────┬─────────────┬──────────────┬─────────────────┐
│ Backup & │ Pilot Light │ Warm Standby │ Hot / Multi- │
│ Restore │ │ │ region active │
├──────────────┼─────────────┼──────────────┼─────────────────┤
│ RTO: hours │ RTO: ~10min │ RTO: minutes │ RTO: seconds │
│ RPO: hours │ RPO: minutes│ RPO: seconds │ RPO: ~0 │
│ $ │ $$ │ $$$ │ $$$$ │
└──────────────┴─────────────┴──────────────┴─────────────────┘
restore scale-up scale-up already live
from cold from seed the warm copy in region BAdda's real numbers
- RPO drives backup frequency. Nightly backup → up to 24h RPO. Hourly snapshots → 1h RPO. Continuous streaming replication → seconds. Each step up in freshness costs Adda more storage, bandwidth, and infra.
- Restore time is real and often huge. Restoring 2TB over a 1 Gbps link is ~4.5 hours at line speed — before decompression, index rebuild, and cache warm-up. If Adda's RTO is 1 hour, a cold restore cannot meet it; you need a warm/hot standby instead.
- Cost ladder. Backup & restore might cost ~5% of primary infra. A hot multi-region active-active setup can cost ~2x Adda's entire production footprint (you run it twice). That's the price of an RTO in seconds — only buy it where the business truly needs it.
- RPO ≈ 0 needs synchronous replication, which adds cross-region write latency (tens of ms per commit). Near-zero data loss and low write latency pull against each other — Adda usually picks one.
When to use it
Backups don't protect against corruption and bad deploys
Redundancy replicates Adda's data instantly — including the DELETE FROM users Shuvo just ran by accident, or ransomware that just encrypted everything. Live replicas faithfully copy the disaster. That's why Adda also needs point-in-time backups it can rewind to a moment before the corruption. DR isn't only about hardware loss; it's about being able to go back in time.
An untested DR plan is a liability, not an asset
The single most common DR failure is discovering, during the real disaster, that the plan doesn't work — the backup was corrupt, the runbook was stale, the standby's capacity was never provisioned. A DR plan's value is exactly equal to how recently Mou tested it. Schedule game days and treat a failed drill as a gift: it's a bug found on Adda's terms instead of the disaster's.
Practice
Recap
- Disaster recovery is about losing a whole site (or corrupting the data itself), not one component — measured by RPO (data you can lose) and RTO (time to recover).
- Strategy is a cost ladder — Backup & Restore → Pilot Light → Warm Standby → Hot/Multi-region — where smaller RTO/RPO costs exponentially more; pick per system by business impact.
- Replicate across regions and keep point-in-time backups (replication copies corruption), and test the plan with game days or your real RTO is unknown.
Redundancy & Failover
Surviving component failure within a site, before you need full DR.
Availability & SLAs
The uptime targets that set how strict your RTO must be.
Replication
The cross-region data copying that shrinks your RPO.
In an interview
Lead with the two numbers: "What's the RPO and RTO for this data?" It instantly frames the conversation and shows you know DR is a business trade-off, not a checkbox. Then map the requirement to a tier — "RPO near zero and RTO in minutes means cross-region replication with a warm standby, not nightly backups" — and always mention two things interviewers love: that replication copies corruption (so you still need point-in-time backups) and that an untested plan has an unknown RTO.
How is this guide?
Last updated on
Backpressure & Circuit Breakers
One slow Adda service starts dragging down everything that calls it. Mou needs a way to fail fast and stop the slowdown from swallowing the whole app.
Design a URL Shortener
Adda is hiring, so the team warms up on a classic: turning a long link into "sho.rt/x7Qa" sounds trivial — until it has to handle a billion links and never hand out the same code twice.