Monitoring & Observability
Adda got slow for hours and nothing paged — because nobody was watching the right thing. Mou can't fix what she can't see, so she gives Adda eyes: logs, metrics, and traces.
The problem
Fahim emails — as always, the canary: "Checkout's been slow all morning." Mou opens Adda; it feels fine to her. She checks the servers; CPU looks normal. Nothing crashed, nothing paged, the failover from last lesson never triggered — because nothing actually died. It just got slow, silently, for hours, and no alert fired. She has no idea if it's one user, one region, one endpoint, or everyone. So she starts guessing, restarting things, and hoping. An hour later she still can't say whether the problem is real, where it lives, or whether her "fix" did anything.
When a distributed system misbehaves — and Adda is a dozen services across three regions now — you can't attach a debugger and step through it. The request touched six services across three machines and it's already over. All that's left are the traces it left behind. If Adda didn't record enough while it ran, the truth is simply gone.
A first attempt
The classic first move, and the one Tanvir reaches for, is to print everything and grep the logs. Sprinkle log lines, ship them to a file, and when something breaks, search.
Logs alone break down fast. They answer "what happened in this one request?" but not "is all of Adda healthy right now?" You can't eyeball a million log lines to know p99 latency doubled. And once Adda is many services, a single tap by Fahim is scattered across many log files with no thread tying them together — you can't reconstruct the one slow request out of the flood. Logs are necessary but they don't scale to system-level questions, and they don't tell you a request's journey.
The insight
Mou needs three different lenses, because "is it healthy?", "what exactly happened?", and "where did the time go?" are three different questions. These are the three pillars of observability:
- Metrics — cheap numeric aggregates over time (request rate, error rate, p99 latency). Answer "is something wrong, and when did it start?"
- Logs — discrete, detailed event records. Answer "what exactly happened in this event?"
- Traces — the path of one request across every service it touched, with timing. Answer "where did the time go?"
Monitoring is watching known signals for known problems. Observability is the richer property Mou is really after: having enough signal to ask new questions about failures nobody anticipated — without shipping new code.
How it works
Emit metrics for the golden signals
Mou instruments every Adda service to count requests, errors, and latency, and expose them for scraping. The four golden signals — latency, traffic, errors, saturation — catch the vast majority of problems with a handful of numbers. Store them in a time-series database so the team can graph and alert on them.
Write structured logs
Log as JSON, not free text: {"level":"error","user":123,"trace_id":"abc","msg":"..."}. Structure lets you filter and aggregate. Crucially, stamp every log with a trace ID so Mou can pivot from a metric spike to the exact logs behind it.
Propagate a trace across services
When a request enters Adda, generate a trace ID and pass it through every downstream call (in a header) — through the API gateway Nabila built, into Orders, Payments, the lot. Each service records a span — its start, end, and metadata — under that shared ID. Stitched together, the spans reconstruct the request's full journey and show which hop was slow.
Alert on symptoms, not causes
Page a human only when users are affected: error rate above the SLO, p99 past its threshold. Don't page on "CPU 80%" — high CPU may be perfectly fine. Alert on the symptom Fahim feels, then use metrics/logs/traces to find the cause.
Build dashboards that tell a story
Arrange panels top-down: overall SLO health, then per-service golden signals, then dependencies. The on-call engineer should read the failure story in 30 seconds, not hunt through 40 unlabeled graphs at 3am.
How the three pillars connect through a shared trace ID:
Adda's real numbers
- Metrics are cheap, logs are expensive. A metric is a few bytes per data point; scraping 1,000 series every 15s is trivial. Adda's logs at 10,000 req/s with 1KB each is 10 MB/s ≈ 864 GB/day — storage and indexing costs dominate, so Mou samples.
- Sample traces. Tracing every request at Adda's QPS is costly, so sample — e.g. 1% head-based, plus keep 100% of errors (tail-based). 1% of 10,000 req/s is still 100 traces/s, plenty to spot patterns.
- Retention tiers. Metrics: keep 13 months at reducing resolution (cheap). Logs: 7–30 days hot, then archive. Traces: hours to days. Match retention to how far back you actually investigate.
- Alert budget. If p99 must stay under 300ms and it's been over for 5 minutes, page. Short windows cause false pages on transient blips; too-long windows burn Adda's error budget before anyone reacts.
When to use it
Alert fatigue is a real outage risk
Every noisy, non-actionable alert trains Adda's on-call to ignore the pager. When the real incident finally fires, it's buried under CPU warnings nobody reads anymore. Mou ruthlessly deletes alerts that don't require human action. A page must mean "a human must do something now" — nothing less.
Cardinality is the silent budget killer
Metrics are cheap until you tag them with high-cardinality labels like user ID or full URL. Each unique label combination is a separate time series; a user_id label can explode 1,000 metrics into millions and melt Adda's monitoring bill. Keep metric labels low-cardinality (status code, region, endpoint template) — put the high-cardinality detail in logs and traces instead.
Practice
Recap
- Three pillars, three questions: metrics ("is it wrong, and when?"), traces ("where did the time go?"), logs ("what exactly happened?").
- A shared trace ID stitched through metrics, traces, and logs is what lets Mou pivot from a spike to the exact failing line.
- Alert on symptoms users feel (errors, latency), keep metric cardinality low, and delete noisy alerts before they cause alert fatigue.
Availability & SLAs
The SLIs and SLOs your metrics are built to measure.
Backpressure & Circuit Breakers
Acting on the saturation and error signals you observe.
Monolith vs Microservices
Why distributed systems make tracing indispensable.
In an interview
When they ask "how do you know the system is healthy?", name the three pillars and, more importantly, tie them to the SLO from the availability lesson: "I'd emit the four golden signals as metrics, alert when error rate or p99 breaches the SLO for 5 minutes, and use trace IDs to pivot into traces and logs to find the cause." Mentioning alert fatigue and metric cardinality signals you've actually operated a system, not just read about one.
How is this guide?
Last updated on
Redundancy & Failover
Everything Adda runs on fails eventually — the question is whether Fahim notices. Mou's answer: a spare that's always ready to take over.
Backpressure & Circuit Breakers
One slow Adda service starts dragging down everything that calls it. Mou needs a way to fail fast and stop the slowdown from swallowing the whole app.