Design a Chat System

The problem

Users have been asking for it for months: they want to talk to each other on Adda, not just post. So Ria greenlights direct messaging, and the team gathers at the whiteboard. The bar Fahim sets is simple to say and hard to build — he types "on my way", hits send, and it should land on his friend's phone before his thumb leaves the screen. His friend is on a train, so their phone flickers between towers, drops off the network for ten seconds, and reconnects — and the message is still there, in order, exactly once. That reliability is the product.

The naive version — the client polls the server every few seconds asking "any new messages?" — technically works and is technically terrible. It's slow (up to a full poll interval of delay), wasteful (millions of empty "nope" responses), and it can't feel instant. Real-time chat forces Adda to invert the flow: the server must push to the client.

Requirements

Functional

• One-to-one messaging, delivered in near real time.
• Message ordering within a conversation.
• Delivery status: sent, delivered, read.
• Online/offline presence.
• Group chat and offline delivery.

Non-functional

• Low latency (< 100 ms end to end when both are online).
• Reliable: exactly-once delivery, no lost messages.
• Highly available and horizontally scalable to hundreds of millions of connections.

A scale estimate

Shuvo sizes the new feature against Adda's user base:

• 500 M daily users, 50 M concurrently connected.
• 40 B messages/day → ~460 K messages/sec average, ~1 M/sec peak.
• Each connection is a long-lived socket → 50 M open sockets. At ~65 K ports per IP and memory per connection, you need thousands of gateway servers just to hold connections open.
• Messages are small (~200 bytes) but numerous — storage grows ~8 TB/day.

The insight

HTTP request/response is pull-shaped — the client asks, the server answers (the same HTTP model from Part 1). Chat needs the opposite: a persistent connection the server can push down at any moment. That's a WebSocket. Once you accept that every online user holds an open socket to some server, Nabila observes, the whole design reorganizes around one question: when A sends to B, how does A's server find B's socket?

How it works

Message ordering

Within a single conversation, order must be stable. Don't trust client clocks. Assign each message a monotonic sequence number per conversation (from the conversation's storage partition or a sequencer). Clients render by sequence, not arrival time, so even out-of-order network delivery displays correctly. Across conversations, global order doesn't matter — which lets you shard by conversation, the same way Shuvo sharded Adda's data in Part 2.

Key decisions and trade-offs

Store messages in a store optimized for the access pattern — "fetch recent messages in a conversation, ordered" — which is a wide-column store (Cassandra-style) partitioned by conversation ID, clustered by sequence number.

Bottlenecks and how to scale

• Connection capacity: add gateway servers; each holds a slice of the 50 M sockets. Presence service is the routing brain and must be fast and replicated.
• Message fan-out in groups: a 1,000-member group turns one send into 1,000 deliveries. This is the same fan-out problem as Adda's feed — deliver to online members via their gateways, queue for offline ones.
• Offline delivery: an inbox/queue per user holds undelivered messages; on reconnect the client syncs everything after its last-seen sequence number.
• Exactly-once: clients attach a client-generated message ID; the server dedupes on it — the same idempotency-key idea from Part 3 — so a retry after a flaky network doesn't create a duplicate.

Practice

Recap

• Chat inverts HTTP's pull model: hold a WebSocket and let the server push.
• A presence/session service routes user → gateway; messages are persisted before delivery and queued when the recipient is offline.
• Order with a per-conversation sequence number, dedupe with client message IDs, and treat group delivery as a fan-out problem.