Message queues

Almost every system grows into queues whether it planned to or not. The pattern shows up the first time somebody on the team realizes the user doesn't actually need to wait for the email to go out before they get the 200 response. You wrap the SMTP call in a setTimeout, ship it, and you've just invented your first queue — an in-memory one, with a delivery guarantee of "if the process doesn't crash", which is exactly as bad as it sounds. The honest version of this is a proper broker (Kafka, RabbitMQ, SQS, Redis Streams), and the moment you adopt one you've signed up for a different way of thinking about correctness.

A queue is not really a buffer. It's a contract between two parts of your system that they will run on different schedules — that producers can keep emitting work without caring whether consumers are awake, and that consumers can take their time without slowing producers down. That contract is what makes the system more available, and it's also what makes everything that depends on the contract subtly harder to reason about. You traded synchronous behavior (request fails immediately if the dependency is down) for asynchronous behavior (request succeeds, but the work might happen later, or twice, or never if you mishandle the dead-letter queue).

Why teams reach for a queue

Three motivations show up over and over, and a real production system usually has all three pulling in the same direction.

The first is decoupling. The producer doesn't need to know who processes its messages, and the consumer doesn't need to know who produces them. You can deploy them on different schedules, scale them independently, replace one without the other noticing. Decoupling is the gift that keeps giving — a year after you add the queue, you can introduce a second consumer group that reads the same stream for analytics, and the original producer doesn't know or care.

The second is smoothing bursty traffic. Real-world inputs are bursty; real-world workers process at a roughly fixed rate. Without a queue, the spike has to be absorbed in-process, which means either dropping requests or building an unbounded in-memory queue (same idea, but unmonitored and able to OOM the host). With a queue, the burst goes into the broker, the workers drain it at their own pace, and the pressure shows up as a depth metric you can graph and alert on instead of a service that fell over.

The third is retry semantics. Once you've got at-least-once delivery and an idempotent consumer, retry is free — the broker re-delivers anything the consumer didn't ack within the timeout. Network blip? Retried. Consumer process crashed mid-handling? Retried. Dependency was down for 30 seconds? Retried after it comes back. You don't write retry logic in the application; the broker handles it, and the only thing you have to get right is making the consumer safe to run more than once.

Kafka or RabbitMQ — the decision that actually matters

These two are not interchangeable. They look similar from a thirty- thousand-foot view ("messages go in one end, come out the other") and they're radically different in what they're good at. Picking wrong adds operational cost forever; picking right is the difference between a queue that you stop thinking about and one that owns a Slack channel.

Kafka is, at its core, a durable, ordered log. Producers append; consumers read by maintaining their own offset; nothing is ever "deleted" until the retention policy expires it. The implication is that multiple independent consumer groups can read the same topic at their own pace — analytics can be 30 seconds behind real-time while the operational consumer is caught up, and that's fine because they're separate readers of the same log. The throughput ceiling is millions of messages per second, and the ordering guarantee is per-partition (within a partition, message order is preserved; across partitions, no guarantee). This makes Kafka the right choice for event-sourcing, real-time analytics pipelines, change-data-capture, and anywhere you might want to replay history later.

RabbitMQ is, at its core, a destination-oriented queue with rich routing. Messages go to a specific destination, consumers compete for them, an acked message is gone. The routing is the star of the show — topic exchanges, fanout, direct, headers — letting you express things like "send this message to all consumers subscribed to orders.*.high-priority" declaratively. Per-message acks, dead-lettering, TTLs, and priority queues are all first-class features. The throughput ceiling is tens of thousands of messages per second, which sounds small next to Kafka but is enormous for the job-queue workloads RabbitMQ is built for.

A useful test: do you want to replay messages later? If yes, Kafka. Do you want rich routing logic at the broker? If yes, RabbitMQ. Most teams who run both eventually settle on Kafka for event streams and RabbitMQ (or its equivalent) for job queues, and that division of labor sticks.

The other contenders show up at the edges. SQS is RabbitMQ's spirit shaped for AWS — fully managed, no broker to run, integrates with everything; choose it when you want zero ops. Redis Streams is the lightweight one — already in your stack if you use Redis, fine throughput, less mature tooling than the giants; choose it when you're early and don't want to add another piece of infrastructure. NATS and Pulsar sit in the gaps and are worth knowing exist; they rarely beat Kafka or Rabbit unless you have a specific reason to look at them.

The dead-letter queue, where systems go to die

After N failed delivery attempts, a broker moves the message to a dead-letter queue and stops retrying. The DLQ exists because some messages are unprocessable — bad data, a schema the consumer doesn't know about, an edge case nobody handled — and infinite retries would just keep crashing your consumers. Better to set the message aside and keep the queue flowing.

The DLQ is the most ignored piece of infrastructure in any queue- based system. Teams set it up, watch it stay empty for months, forget it exists, and then learn it has 14,000 orders in it the day a customer complains. The fix isn't more clever code; it's an alert on DLQ depth. If the DLQ is non-empty, someone should be paged. If the DLQ is growing, something is currently broken and you have hours, not weeks, to fix it.

A DLQ that nobody monitors is just data loss with extra steps. The alert that catches DLQ growth has saved more outages than any amount of consumer-side error handling.

The triage discipline matters. When the DLQ fills up, you have to decide per message: is this a bug we can fix and replay (most common), bad data we can drop (less common), or something that needs a human decision (rare but real). Production teams build small admin tools that let them inspect the message, see the exception, decide what to do, and either replay it or drop it. The tool is unglamorous and load-bearing.

Ordering, the assumption that breaks teams

The other thing that surprises people in production is what their broker actually guarantees about message order. Kafka guarantees order within a partition. RabbitMQ guarantees order within a queue, until a redelivery scrambles things. Neither guarantees order across the broker as a whole, and there's no efficient way to get that — global ordering would require a single bottleneck.

The way out is to design so you don't need global ordering. Partition by the key whose ordering matters (user_id, order_id) so related events stay on the same partition and arrive in order. Anything that needs cross-key ordering — "this email must go out after that database write" — needs a different tool than a vanilla queue: a saga, an outbox pattern, or making the events causally self-describing so the consumer can reorder. See [CONCEPT]outbox-pattern for the canonical fix.

How much lag does a burst actually buy you?

Queue depth feels abstract until you turn it into wall-clock delay. The math is a single division: a backlog drains at (consumer_rate − arrival_rate) messages per second, and the time-to-drain is depth / drain_rate. Run it before you trust a queue to absorb a spike.

Say steady state is 200 msg/sec and your consumer fleet tops out at 1,000 msg/sec. A promo drives a burst:

So a 90-second spike leaves messages waiting up to ~13 minutes. If the side effect is a welcome email, fine. If it's a 5-minute coupon or a one-time login code, you just shipped expired credentials at scale. The number that matters is not throughput — it's oldest-message age, and you should graph and alert on it directly.

A failure I'll never forget

A team I worked with had a Kafka pipeline that consumed user signups and sent welcome emails. The consumer was simple, the broker was healthy, throughput was a steady 200 messages/sec. On Black Friday they ran a promo and signups spiked to 8,000 messages/ sec for 90 seconds. The broker absorbed it fine. The consumer couldn't keep up, queue depth grew to 380,000 messages, then slowly drained over the next 25 minutes.

The bug nobody noticed was that the welcome email had a 24-hour TTL on a coupon. By the time the slowest emails went out — five minutes late — they were still valid. By the time the team rolled out a similar promo two months later with a 5-minute coupon, the same queue depth meant the emails arrived after the coupons expired. The team got bug reports for two days before someone correlated it with queue lag.

The lesson is that queue lag is invisible until the downstream consequence makes it visible. Always alert on lag, not just on throughput. Always think about what "five minutes late" means to the person receiving the eventual side effect. And if the side effect is time-sensitive (a coupon, a verification code, a real-time notification), you may need to defer to a different mechanism entirely — push notifications, an inline render, a shorter freshness contract.

When to walk away

• Strict request/response semantics where the caller needs the result — use RPC, not a queue. A queue is a fire and check later primitive.
• Strict global ordering across all keys — most brokers only do per-partition order. Cross-partition ordering means single- threading or careful sequencing on the consumer.
• "Just a 5-second delay queue" — Postgres plus a worker that polls every second is often simpler than introducing a broker.
• Hard real-time — a queue adds milliseconds at best, hundreds of milliseconds typically. If you need sub-10ms guaranteed, you need in-process handling.

[CONCEPT]circuit-breaker is what protects you when the consumer's own dependencies start to fail. [CONCEPT]event-sourcing is what you build on top of Kafka once you've decided the log itself is the source of truth.