Message Queues
A video attached to an Adda post takes 90 seconds to process — and it's holding the upload request hostage. When Adda goes viral, that one slow task takes down the whole API. A queue absorbs the shock.
The problem
A user attaches a video to an Adda post. The Posts service has to transcode it into five resolutions, generate thumbnails, scan it for copyright, and notify the user when it's done — maybe 90 seconds of work. Right now the upload request does all of it inline, so the user's app sits spinning for a minute and a half, and if anything fails, the whole upload errors out.
Then Adda trends. A thousand video uploads land in one minute. Each ties up a request thread for 90 seconds, the thread pool is exhausted in seconds, and now unrelated requests — logins, feed loads, plain text posts — start timing out too. Fahim, the canary as ever, can't even open his feed. One slow task took down the whole API.
A first attempt
Tanvir's first move: add more workers and bigger machines so Adda can process everything the instant it arrives. Provision for the peak.
But the peak is the problem. Adda's traffic isn't smooth — it spikes when a post goes viral. If you size for the worst minute of the year, you pay for idle capacity the other 525,599 minutes. Size for the average and the spikes knock you over. And the deeper issue remains: the producer (uploads) and the consumer (transcoding) are locked together in time. The upload cannot finish until transcoding does. You need them to run at their own speeds.
The insight
Put a buffer between them. The Posts service drops a "transcode this video" message into a queue and immediately returns "got it, we'll notify you." Separate worker processes pull messages off the queue and transcode at whatever rate they can sustain.
Now producer and consumer are decoupled in time. Uploads finish in milliseconds. A spike just makes the queue longer, not the API slower — the queue absorbs the shock and drains when the surge passes. Add workers to drain faster; the API never notices.
A queue trades latency for resilience
By accepting work now and doing it later, a queue turns a synchronous, all-or-nothing chain into an asynchronous one that bends under load instead of breaking. The cost: the result isn't ready the instant the request returns.
How it works
Producer enqueues and moves on
The Posts service validates the request, writes a message (video ID, options) to the queue, and returns immediately. Its job is done in a few milliseconds.
The queue holds the backlog
Messages wait durably in order of arrival. If 1,000 arrive in a burst, all 1,000 sit safely in the queue — nothing is dropped, nothing blocks the producer.
Consumers pull at their own pace
Worker processes each grab a message, process it, and ask for the next. Slow consumers just leave more in the queue; Adda scales throughput by adding workers.
Acknowledge, retry, or dead-letter
A worker acks only after success, so the message is removed. If it crashes mid-job, the un-acked message reappears for another worker. After N failed tries, the message goes to a dead-letter queue for a human to inspect instead of retrying forever.
Concrete numbers
Enqueuing a message costs ~1–5 ms; the user sees that instead of 90 seconds. A broker like RabbitMQ or SQS sustains tens of thousands of messages per second per queue, and a partitioned log like Kafka pushes into the millions.
The buffering math: say Adda's uploads spike to 1,000/min for 5 minutes (5,000 messages) while each worker transcodes one video every 90 s (~0.67/min). With 20 workers you drain ~13/min, so the backlog clears in a few minutes after the spike — users wait a little longer for their "done" notification, but nothing fails and the API never stalls. Without the queue, those 5,000 jobs would have exhausted the thread pool and taken the whole service down — exactly the outage that started this lesson.
When to use it
At-least-once means duplicates — design for them
Most queues guarantee at-least-once delivery: if a worker processes a message but crashes before acking, the message is redelivered and processed again. "Exactly-once" is largely a marketing promise; in practice you get at-least-once and make your consumers idempotent so a duplicate is harmless (transcode once if not already done for this message ID). Adda meets that problem head-on a few lessons from now.
Queues hide overload — watch the depth
A queue absorbing a spike is healthy; a queue that grows without ever draining means Adda's consumers are permanently too slow, and the "buffer" is now just a delay machine hiding a capacity problem. Alert on queue depth and age-of-oldest-message, not just on errors — a silently growing backlog can mean hours-old data by the time anyone notices.
Practice
Recap
- A queue decouples producer and consumer in time: the producer enqueues and returns fast, consumers process at their own pace, and spikes grow the backlog instead of breaking the API.
- Delivery is at-least-once, so duplicates happen — make consumers idempotent, ack only after success, and send repeated failures to a dead-letter queue.
- The queue hides overload, so monitor depth and message age; a backlog that never drains is a capacity problem wearing a buffer's clothes.
Publish / Subscribe
When one Adda event needs to reach many services, not one worker.
Idempotency
Making at-least-once delivery safe by tolerating duplicates.
Backpressure & Circuit Breakers
What to do when the backlog keeps growing.
In an interview
How to discuss this
Reach for a queue whenever work can be done asynchronously — "the API accepts the job and a worker processes it later." Say the two words that show you've operated one: at-least-once (so consumers must be idempotent) and dead-letter queue (so failures don't loop forever). Mention monitoring queue depth as your overload signal, and be ready to contrast point-to-point queues with broadcast pub/sub.
How is this guide?
Last updated on
The API Gateway
Adda's monolith is now a dozen services, and every one of them re-checks the auth token. Nabila's first week: build the single front door that does routing, auth, and rate limits once.
Publish / Subscribe
One new Adda post must ripple to the feed, search, notifications, and analytics. If the Posts service has to call each of them, it becomes everyone's bottleneck. Pub/sub lets it broadcast and forget.