4 ms·

This is asking for a huge favor, but can anyone explain to me what Kafka does that RabbitMQ doesn't? Is it just scale?

wmal 3y ago
They are conceptually different. Kafka optimizes throughput of data. I wouldn't use Kafka for a job queue, and wouldn't use RabbitMQ for streaming data when ordering would be important.
leetharris 3y ago
They are very different, but some people use them for similar things. Kafka is a stream, RabbitMQ is a queue. Without getting into the details, RabbitMQ is designed to add things to a stack and pop them off when consumed. Kafka is designed to stream everything to a continuous log and anywhere can tune in when appropriate.
flashgordon 3y ago
One way to think about it is who does the routing. My understanding of rabbitmq from 10 years ago was that RMQ pushes to connected consumers and to only 1 at a time? You'd need a fanout consumer that does more of the work. And lower throughput overall. With Kafka it "just" keeps appending to a dumb (but huge) circular buffer. But you can have multiple consumers read off this buffer and can start any point. Downside is customers have to maintain their own offsets (in some storage) but there is now a big decoupling between producer and consumer. This contributes a large part to high throughput too (and consumers can go at their own pace -ofcourse if they are too slow they can fall off the log).
- teraflop 3y ago
  > Downside is customers have to maintain their own offsets (in some storage) Minor correction: you can maintain the offsets yourself if you want, but usually it's not necessary because Kafka can do it for you. The abstraction Kafka provides is that for each consumer group and for each (topic, partition) tuple, your consumer object that is guaranteed not to receive messages before the last offset at which you called commit(). Internally, the committed offsets are stored in a special Kafka topic of their own.
  - flashgordon 3y ago
    Ah is this offset maintained by the Kafka client? (I included that as "client" as well). I thought the Kafka topic itself did not maintian any client specific offsets (how could it)? Unless this was added in recent years. Interesting tho!
facorreia 3y ago
This blog post does a good job of explaining how they compare on different use cases: https://eranstiller.com/rabbitmq-vs-kafka-an-architects-dilemma-part-2 https://eranstiller.com/rabbitmq-vs-kafka-an-architects-dile...
RomanPushkin 3y ago
Kafka events are replayable, and represent the final state we're aiming for. You might have two exactly the same events that tell you how the state should look like. And there are batches of events. It's good for batch processing - "we're getting new portion of data to train AI model". RabbitMQ messages are supposed to be processed/consumed/acked only once. Your app most probably won't ever get two exactly the same messages, unless you misconfigured/misued RabbitMQ. It's good for classic message processing - "used clicked something, run the job of informing subscribers that new post has been created" (because you can't send 1000 messages from a web worker thread).
- bikezen 3y ago
  This hasnt been true for awhile, RabbitMQ added streams back in 2021.

wmal 3y ago

They are conceptually different. Kafka optimizes throughput of data. I wouldn't use Kafka for a job queue, and wouldn't use RabbitMQ for streaming data when ordering would be important.

leetharris 3y ago

They are very different, but some people use them for similar things. Kafka is a stream, RabbitMQ is a queue. Without getting into the details, RabbitMQ is designed to add things to a stack and pop them off when consumed. Kafka is designed to stream everything to a continuous log and anywhere can tune in when appropriate.

flashgordon 3y ago

One way to think about it is who does the routing. My understanding of rabbitmq from 10 years ago was that RMQ pushes to connected consumers and to only 1 at a time? You'd need a fanout consumer that does more of the work. And lower throughput overall. With Kafka it "just" keeps appending to a dumb (but huge) circular buffer. But you can have multiple consumers read off this buffer and can start any point. Downside is customers have to maintain their own offsets (in some storage) but there is now a big decoupling between producer and consumer. This contributes a large part to high throughput too (and consumers can go at their own pace -ofcourse if they are too slow they can fall off the log).

teraflop 3y ago
> Downside is customers have to maintain their own offsets (in some storage) Minor correction: you can maintain the offsets yourself if you want, but usually it's not necessary because Kafka can do it for you. The abstraction Kafka provides is that for each consumer group and for each (topic, partition) tuple, your consumer object that is guaranteed not to receive messages before the last offset at which you called commit(). Internally, the committed offsets are stored in a special Kafka topic of their own.
- flashgordon 3y ago
  Ah is this offset maintained by the Kafka client? (I included that as "client" as well). I thought the Kafka topic itself did not maintian any client specific offsets (how could it)? Unless this was added in recent years. Interesting tho!

facorreia 3y ago

This blog post does a good job of explaining how they compare on different use cases: https://eranstiller.com/rabbitmq-vs-kafka-an-architects-dilemma-part-2 https://eranstiller.com/rabbitmq-vs-kafka-an-architects-dile...

RomanPushkin 3y ago

Kafka events are replayable, and represent the final state we're aiming for. You might have two exactly the same events that tell you how the state should look like. And there are batches of events. It's good for batch processing - "we're getting new portion of data to train AI model". RabbitMQ messages are supposed to be processed/consumed/acked only once. Your app most probably won't ever get two exactly the same messages, unless you misconfigured/misued RabbitMQ. It's good for classic message processing - "used clicked something, run the job of informing subscribers that new post has been created" (because you can't send 1000 messages from a web worker thread).

bikezen 3y ago
This hasnt been true for awhile, RabbitMQ added streams back in 2021.