4 ms·
The biggest difference, IMO, is that Kafka is typically used when a message will be consumed by multiple consumers, whereas RabbitMQ or SQS generally send a mes
by saryant 11y ago
The biggest difference, IMO, is that Kafka is typically used when a message will be consumed by multiple consumers, whereas RabbitMQ or SQS generally send a message to a single consumer.
We use it to ingest ~40mb/s and fan it out to a number of consuming applications.
I'll also add that if you put some thought behind your topic replication and partitioning you can build some incredibly resilient applications. Also that "immutable" isn't necessarily true, it's common for Kafka topics to roll off messages based on time or size. (That's just to clarify for those not familiar with Kafka. I realize that you mean messages are not deleted or modified once written to a topic, other than by topic retention settings)
- manigandham 11y agoIt's very easy to setup multiple consumers with RabbitMQ topics though and SQS is a very basic queueing system that just doesn't support much. To me, Kafka is just meant for much larger magnitudes of scale and persistence (of the entire log of messages for however long you need) as a core feature. Google's PubSub is still the best blend of traditional queue semantics with Kafka scale and persistence though.
- asymmetric 11y agoIt's incorrect to say that RMQ sends message to consumers. What it does is it routes messages to queues/exchanges. It's then entirely up to you to decide how many consumers will effectively consume them, I.e. You can have as many consumers as you wish.
- dkersten 11y agomessages are not deleted or modified once written to a topic, other than by topic retention settings Except if you have log compaction turned on, I guess. A key "selling point" of Kafka for me is that each consumer can decide from when they wish to receive messages. That is, you can replay the messages.