Message Queues: The Shock Absorbers of Distributed Systems
When to reach for a queue, how the ack loop actually works, and how partitioning scales consumption - without creating hot partitions.
On this page 4 sections
Every distributed system eventually meets the same moment: traffic arrives faster than the server can process it. You can scale the server, or you can change the shape of the problem. A message queue does the latter - and it is one of the highest-leverage tools in systems design.
When a queue earns its place
Reach for a message queue when any of these is true:
- High incoming throughput. Requests arrive faster than the downstream work can complete. Instead of dropping or blocking, producers write to the queue and move on.
- Bursty traffic. This is the classic case. A flash sale, a batch import, a webhook storm - the queue absorbs the spike, and consumers drain it at their own steady pace. Your workers are sized for average load, not peak load.
- Async processing. Not everything needs to happen in the request path. Sending an email, generating a report, updating a search index - enqueue it, return to the caller, process later.
- Decoupling. The producer does not know or care who consumes the message, or whether the consumer is even up right now. Each side deploys, scales, and fails independently.
Conceptually, a queue is a simple thing: a logical unit where messages are grouped together, waiting to be processed.
The core loop: fetch, process, ack
The lifecycle of a message is a contract between the queue and the consumer:
- A producer writes the message to the queue.
- A consumer fetches it and hands it to a worker for processing.
- Once processing succeeds, the consumer acknowledges the message back to the queue.
- Only on ack is the message removed from the queue.
That last step is the important one. If a worker crashes mid-processing, no ack is sent, and the queue redelivers the message to another worker. Nothing is lost - but it also means a message can be delivered more than once. This is at-least-once delivery, and it has a design consequence: make your processing idempotent, so handling the same message twice produces the same result as handling it once.
Scaling: partitions, and the hot-partition trap
One queue with one consumer eventually becomes the new bottleneck. The scaling approach is
partitioning: split the queue into partitions, route each message to a partition by a key
(say, hash(user_id)), and let the consumer group allocate different workers to fetch from
different partitions. Three partitions, three workers, three parallel streams - ordering is
preserved within each partition, and throughput scales with the partition count.
The trap to watch for is the hot partition. Partitioning only helps if load spreads evenly.
Pick a bad key - say, customer_id when one customer generates half your traffic - and one
partition takes the flood while the others idle. Your “scaled” system is now exactly as fast as
its busiest partition. Choose keys with high cardinality and even distribution, and measure
per-partition lag, not just overall queue depth.
The takeaway
A message queue is not just infrastructure - it is a design decision about time. It converts “do this now, at the caller’s pace” into “do this reliably, at the system’s pace.” For bursty, high-throughput, or async workloads, that conversion is usually the difference between a system that degrades gracefully and one that falls over.