Message Queue vs Stream
What it is
- Message queue (mental model: SQS, RabbitMQ classic queues): producers send messages; consumers pull or receive push; messages are often deleted after successful processing. Good for job dispatch and decoupling services with at-least-once delivery.
- Event stream / log (mental model: Kafka, Pulsar, Kinesis): messages are append-only partitions; consumers track offsets and can replay history. Good for analytics, event sourcing, and multiple independent consumer groups.
When to use
| Need | Favor |
|---|---|
| Simple task queue, per-message visibility timeout | Queue (SQS-like) |
| Fan-out to many services, replay, stream processing | Kafka-like stream |
| Strict ordering for one entity | Single partition key / single shard |
| Global ordering | Hard—usually avoid; partition by key |
Ordering guarantees
- Single partition / single queue with one consumer: FIFO processing (often exactly one effective consumer at a time).
- Multiple consumers: ordering not global unless you assign partition key so related events share one partition (Kafka) or use FIFO queues with limited throughput (AWS FIFO).
- Queues: ordering between unrelated messages is weak; visibility timeout can cause redelivery and reordering perception.
Loading diagram…
Alternatives
- Synchronous RPC for critical path when you need immediate consistency (simpler operationally, tighter coupling).
- Database outbox pattern: transactional writes + relay to broker—stronger than “fire and forget”.
Failure modes
- Poison messages: infinite retries; use DLQ (dead-letter queue) and alerts.
- Consumer lag: scale consumers; partition bottleneck for streams.
- Duplicate processing: design idempotent handlers (at-least-once is default for many systems).
- Back-pressure: unbounded queues hide overload until catastrophic lag—monitor depth.
Interview talking points
- Clarify delivery semantics: at-most-once vs at-least-once vs exactly-once (often “exactly-once” is effectively via idempotency + offsets).
- Compare ordering scope vs throughput (more partitions → more parallelism, weaker global order).
- Mention replay for streams vs delete-on-ack for queues.
- Relate consumer throughput and end-to-end latency to latency and throughput.
Related fundamentals
- Back-of-the-envelope estimation — queue depth, throughput, consumer count
- Latency and throughput
Last updated on
Spotted something unclear or wrong on this page?