5 ms·
One way to think about it is who does the routing. My understanding of rabbitmq from 10 years ago was that RMQ pushes to connected consumers and to only 1 at a
by flashgordon 3y ago
One way to think about it is who does the routing. My understanding of rabbitmq from 10 years ago was that RMQ pushes to connected consumers and to only 1 at a time? You'd need a fanout consumer that does more of the work. And lower throughput overall.
With Kafka it "just" keeps appending to a dumb (but huge) circular buffer. But you can have multiple consumers read off this buffer and can start any point. Downside is customers have to maintain their own offsets (in some storage) but there is now a big decoupling between producer and consumer. This contributes a large part to high throughput too (and consumers can go at their own pace -ofcourse if they are too slow they can fall off the log).
- teraflop 3y ago> Downside is customers have to maintain their own offsets (in some storage) Minor correction: you can maintain the offsets yourself if you want, but usually it's not necessary because Kafka can do it for you. The abstraction Kafka provides is that for each consumer group and for each (topic, partition) tuple, your consumer object that is guaranteed not to receive messages before the last offset at which you called commit(). Internally, the committed offsets are stored in a special Kafka topic of their own.
- flashgordon 3y agoAh is this offset maintained by the Kafka client? (I included that as "client" as well). I thought the Kafka topic itself did not maintian any client specific offsets (how could it)? Unless this was added in recent years. Interesting tho!