4 ms·

It's a message queue. You use it for everything you want to do outside of the general request cycle. IE: Making API calls, priming cache, sending emails, etc..

It's a message queue. You use it for everything you want to do outside of the general request cycle. IE: Making API calls, priming cache, sending emails, etc.. Biggest competitors of Kafka are RabbitMQ and amazon SQS.

manigandham 11y ago
It's not a message queue, it's a logging system. Queues are meant for ephemeral messages that expire once consumed. Kafka is immutable log storage that can be read as many times as necessary by consumers. Biggest competitors would be AWS Kinesis, Azure EventHubs and Google PubSub.
- saryant 11y ago
  The biggest difference, IMO, is that Kafka is typically used when a message will be consumed by multiple consumers, whereas RabbitMQ or SQS generally send a message to a single consumer. We use it to ingest ~40mb/s and fan it out to a number of consuming applications. I'll also add that if you put some thought behind your topic replication and partitioning you can build some incredibly resilient applications. Also that "immutable" isn't necessarily true, it's common for Kafka topics to roll off messages based on time or size. (That's just to clarify for those not familiar with Kafka. I realize that you mean messages are not deleted or modified once written to a topic, other than by topic retention settings)
  - manigandham 11y ago
    It's very easy to setup multiple consumers with RabbitMQ topics though and SQS is a very basic queueing system that just doesn't support much. To me, Kafka is just meant for much larger magnitudes of scale and persistence (of the entire log of messages for however long you need) as a core feature. Google's PubSub is still the best blend of traditional queue semantics with Kafka scale and persistence though.
  - asymmetric 11y ago
    It's incorrect to say that RMQ sends message to consumers. What it does is it routes messages to queues/exchanges. It's then entirely up to you to decide how many consumers will effectively consume them, I.e. You can have as many consumers as you wish.
  - dkersten 11y ago
    messages are not deleted or modified once written to a topic, other than by topic retention settings Except if you have log compaction turned on, I guess. A key "selling point" of Kafka for me is that each consumer can decide from when they wish to receive messages. That is, you can replay the messages.

manigandham 11y ago

It's not a message queue, it's a logging system. Queues are meant for ephemeral messages that expire once consumed. Kafka is immutable log storage that can be read as many times as necessary by consumers. Biggest competitors would be AWS Kinesis, Azure EventHubs and Google PubSub.

saryant 11y ago
The biggest difference, IMO, is that Kafka is typically used when a message will be consumed by multiple consumers, whereas RabbitMQ or SQS generally send a message to a single consumer. We use it to ingest ~40mb/s and fan it out to a number of consuming applications. I'll also add that if you put some thought behind your topic replication and partitioning you can build some incredibly resilient applications. Also that "immutable" isn't necessarily true, it's common for Kafka topics to roll off messages based on time or size. (That's just to clarify for those not familiar with Kafka. I realize that you mean messages are not deleted or modified once written to a topic, other than by topic retention settings)
- manigandham 11y ago
  It's very easy to setup multiple consumers with RabbitMQ topics though and SQS is a very basic queueing system that just doesn't support much. To me, Kafka is just meant for much larger magnitudes of scale and persistence (of the entire log of messages for however long you need) as a core feature. Google's PubSub is still the best blend of traditional queue semantics with Kafka scale and persistence though.
- asymmetric 11y ago
  It's incorrect to say that RMQ sends message to consumers. What it does is it routes messages to queues/exchanges. It's then entirely up to you to decide how many consumers will effectively consume them, I.e. You can have as many consumers as you wish.
- dkersten 11y ago
  messages are not deleted or modified once written to a topic, other than by topic retention settings Except if you have log compaction turned on, I guess. A key "selling point" of Kafka for me is that each consumer can decide from when they wish to receive messages. That is, you can replay the messages.