Design a News Feed
This is the payoff: Adda's own feed, at the scale it has finally reached. Showing fresh posts from hundreds of people, instantly, to 300 million users — the fan-out problem has no perfect answer.
The problem
This is the one the whole book has been building toward. Adda is a feed — it always was, back when Fahim's "first post lol" landed on Ria's laptop. Now Fahim opens the app and, in under a second, sees the latest posts from the 500 people he follows, newest-ish first, mixed and ranked. It feels effortless. Behind that second, Adda had to gather posts scattered across 500 different authors, some of whom posted while Fahim was asleep, and assemble them into one list — for him, and for 300 million other people, each with a different 500.
The version Ria first shipped — "when Fahim opens his feed, query every account he follows for their recent posts and merge" — was a beautiful idea that died the week Adda crossed a few million users. Mou was paged at 3am for feed reads timing out; the database was drowning. The hard part of a feed, the team now knows, isn't storing posts. It's the fan-out: matching a flood of writes to a flood of reads.
Requirements
Functional
- Publish a post.
- Fetch a user's home feed (posts from people they follow), paginated.
- Feed is roughly reverse-chronological, optionally ranked.
Non-functional
- Feed load must be fast (< 200 ms) — this is the read hot path.
- Eventual consistency is fine; a post appearing a few seconds late is acceptable.
- Handle highly skewed follower counts (Fahim follows 500; an Adda celebrity has 50 M followers).
A scale estimate
Shuvo runs the numbers on Adda's real traffic:
- 300 M daily active users, each loading the feed ~10×/day → ~35 K feed reads/sec average, ~100 K/sec peak.
- 2 M posts/sec is unrealistic; assume ~5 K posts/sec.
- Average user follows ~200 accounts; average post has ~200 followers to reach.
- Total fan-out work per post is "post rate × avg followers" — cheap on average, brutal in the tail.
Reads outnumber writes ~20:1, but a single write can generate millions of feed updates. That asymmetry is the entire design tension.
The two approaches
There are exactly two places Adda can do the merging work, Tanvir points out: at write time or at read time.
Fan-out on read (pull). Store each post once. When a user opens their feed, query all the people they follow and merge on the fly. Writes are trivial; reads are expensive — merging 500 authors' timelines every single load, 100 K times a second. This is exactly the version that paged Mou. Feed latency suffers.
Fan-out on write (push). When someone posts, immediately copy that post's ID into the precomputed feed of every follower. Reads become a single lookup ("give me my feed list"). But an Adda celebrity posting to 50 M followers triggers 50 M writes — a fan-out storm.
The insight
Neither pure approach wins, and the team stops looking for the one that does. The trick is a hybrid: push for ordinary users (fast feeds, cheap writes), pull for the handful of celebrities (avoid the storm), and merge the two at read time.
How it works
Write a post
Persist the post once in a posts store. Emit a "new post" event onto the message queue — the same queue the team added in Part 3 — so fan-out happens asynchronously. The author's request returns immediately.
Fan out to followers (push path)
A worker reads the event and, for each follower, appends the post ID to that follower's feed list in a fast store (Redis list per user, capped at ~800 entries). Ordinary authors with thousands of followers fan out in the background within seconds.
Skip the storm for celebrities
If an author's follower count exceeds a threshold (say 100 K), do not fan out. Their posts stay only in the posts store and are pulled at read time.
Assemble the feed (read path)
On feed load: read the user's precomputed feed list (push results), then pull recent posts from the few celebrities they follow, merge, rank, and paginate. One cheap list read plus a handful of pulls — not 500.
Key decisions and trade-offs
Hybrid fan-out is the standard answer
Push gives readers O(1) feeds; pull gives writers relief from the celebrity storm. Splitting on follower count captures the best of both. The cost is complexity — Adda now runs two code paths and merges them — but it's the design real systems (Twitter's timeline) converged on.
Precomputed feeds can go stale
Push-based feeds are denormalized copies. If someone unfollows, or a post is deleted, those already-fanned-out entries are now wrong. Filter deleted/blocked content at read time, and accept that feeds are eventually consistent rather than trying to rewrite millions of lists.
Ranking (engagement, recency, affinity) happens at read time on the merged candidate set — keep it a bounded list (a few hundred candidates) so scoring stays cheap.
Bottlenecks and how to scale
- Fan-out storms: the hybrid threshold is the primary defense. Fan-out workers scale horizontally off the queue, and slow followers don't block the author.
- Feed cache memory: cap each feed at ~800 IDs and store only IDs, not full posts — hydrate post bodies from a cache on read.
- Hot celebrity posts: these are pulled by millions; cache the celebrity's recent-posts list hard, close to the edge.
- Read fan-in: limit how many celebrities you pull per load; if a user follows 50 celebrities, batch the pulls and cache the merged result briefly.
Practice
Recap
- Adda's feed is a fan-out problem: reconciling many writes with many reads.
- Push (fan-out on write) makes reads cheap; pull avoids the celebrity storm; the hybrid does both and merges.
- Store IDs not bodies, fan out asynchronously via a queue, and rank a bounded candidate set at read time.
Message Queues
Decoupling the post write from the fan-out work.
Caching Fundamentals
Hydrating post bodies and hot celebrity timelines.
Design a Chat System
Adda's next feature — real-time delivery with strict ordering.
In an interview
Name both approaches, state their costs in one breath ("push = cheap reads, expensive writes; pull = the reverse"), then propose the hybrid and define the threshold. If you only remember one thing, remember the celebrity fan-out storm — being able to explain why it happens and how the split fixes it is what separates a strong answer from a textbook one.
How is this guide?
Last updated on
Design a URL Shortener
Adda is hiring, so the team warms up on a classic: turning a long link into "sho.rt/x7Qa" sounds trivial — until it has to handle a billion links and never hand out the same code twice.
Design a Chat System
Adda adds real-time messaging: messages that arrive instantly, in order, even when a phone drops off the network. Deceptively deep — and it stretches everything the team learned.