Message Queues vs. Kafka: Core Models

"Just put it on a queue" and "just put it on Kafka" sound like the same advice. They're not. A traditional message queue and Kafka's log solve overlapping but distinct problems, and reaching for the wrong one costs you either unnecessary operational weight or a system that quietly can't do what you need six months from now. The difference comes down to one design decision: what happens to a message after it's been read.

The traditional queue: deliver, then delete

Systems like RabbitMQ and Amazon SQS implement the model most engineers learn first. A producer pushes a message onto a queue. A consumer pulls it off, processes it, and acknowledges it — at which point the broker deletes the message. It's gone. If a second consumer wanted to look at that same message, it's too late.

This is a perfect fit for work distribution. If you have ten thousand images to resize, you don't want ten thousand workers each doing the full job — you want each image resized exactly once, by whichever worker happens to be free. Queues make this trivial: point a pool of competing consumers at the same queue, and the broker hands each message to only one of them. Add more workers, and throughput scales horizontally with almost no extra code. The queue itself tracks very little state — mostly "has this message been acknowledged yet" — which is exactly why these brokers are simple to reason about and cheap to run.

The cost of that simplicity is that the queue has no memory. Once a message is consumed, there is no going back to replay it, no way for a newly deployed service to catch up on history it missed, and no way for two unrelated teams to independently process the same stream of events. The queue is a to-do list, not a record.

Kafka's log: append, retain, never delete on read

Kafka starts from a different primitive entirely: the durable, append-only log. A producer writes a message to a topic, which is really an ordered sequence of records split across one or more partitions. That record is appended to the end of its partition's log and stays there for a configurable retention period — hours, weeks, or indefinitely — regardless of whether anyone has read it yet.

Consumers don't pull a message and remove it. Instead, each consumer tracks an offset — a position in the log — and reads forward from there. Reading a record doesn't touch it; it just advances the reader's own bookmark. The broker doesn't know or care who has "consumed" a message in the queue sense, because from Kafka's point of view nothing is ever consumed — it's just read.

Fan-out: one reader vs. many independent readers

This difference in retention drives a completely different fan-out story. In a queue, the moment one consumer takes a message, it's unavailable to every other consumer — that's the whole point, it's how you avoid double-processing work. If you want three different services to react to the same event, you typically need three separate queues (often fanned out from a topic exchange or SNS topic), each with its own copy of the message.

Kafka inverts this. Any number of consumer groups can read the same topic independently, each maintaining its own offset into the log. Your fraud-detection service, your analytics pipeline, and your notification service can all read the same stream of order events, at their own pace, without knowing about each other or stepping on each other's progress. A consumer group that's down for an hour simply resumes from its last committed offset — no messages are lost, because nothing was ever removed.

Parallelism still happens, but at a different layer. Within a single consumer group, Kafka divides a topic's partitions among the group's consumer instances — each partition is read by exactly one consumer in the group at a time. That's how you get the queue-like "spread the work across workers" behavior and keep multiple independent groups reading the full history, at the same time, on the same topic.

Ordering: guaranteed within a partition, not across a topic

Kafka guarantees strict ordering only within a single partition. Across a topic's partitions, there is no global ordering guarantee at all — messages in partition 0 and messages in partition 1 can be interleaved in any order relative to each other, because they're independent logs written and read in parallel.

This makes your partition key one of the most consequential decisions in a Kafka design. Kafka hashes the key you provide to pick a partition, so every message with the same key always lands on the same partition — and therefore is always read back in the order it was written, relative to other messages with that key.

// Bad: round-robin key, no ordering guarantee across a user's events
producer.send("user-events", randomKey(), event);

// Good: same user always lands on the same partition,
// so their events are read back in the order they happened
producer.send("user-events", event.userId, event);

For ordering-sensitive data — a user's account events, an order's lifecycle transitions, a device's telemetry — key by the entity whose history must stay ordered (user ID, order ID, device ID). Get this wrong and you'll see it as a rare, maddening bug: a "cancelled" event processed before the "created" event it followed, simply because the two landed on different partitions and were read out of sequence.

When a plain queue is still the right call

None of this makes Kafka a strict upgrade. Running a Kafka cluster — or even a managed one — is meaningfully more operational surface than a managed queue, and most of that machinery is wasted if all you need is reliable delivery of discrete tasks. Background job processing, sending a transactional email, resizing an uploaded file, retrying a flaky API call with backoff — these are all "do this thing once, then forget it" problems. A queue's built-in visibility timeouts, dead-letter queues, and per-message retry semantics solve exactly that, with far less to operate and reason about.

The question worth asking before reaching for Kafka is simple: do you need more than one independent consumer to read this data, or do you need to replay history that's already been processed? If the honest answer is no — if there's exactly one thing that needs to happen to each message, once — a queue will get you there faster, cheaper, and with far fewer moving parts to keep alive.