5 ms·
Besides the operational concerns mentioned in the sibling comment, Kafka is simply not a great queue. You can't work-steal, you can't easily retry out-of-order,
by morelisp 3y ago
Besides the operational concerns mentioned in the sibling comment, Kafka is simply not a great queue. You can't work-steal, you can't easily retry out-of-order, you can't size retention based on "is it processed yet", and you may need to manually implement DLQ behavior.
If you already have Kafka for other (more WAL-y, or maybe older log-shippy) reasons it can be an OK queue, especially if you've got a team that can use Kafka as a WAL they can easily work around using most of the downsides of using it as queue. But I wouldn't take it as a first choice.
- parhamn 3y agoGreat point. The basic semantics are very different too. In MQs you partition/namespace/channel (whatever you want to call it) based on how data flows in your application (e.g. fanout). In Kafka you're tied more to a persistance model so you end up with fat linear topics and the "filtering"/flow management happen on the consumer's side.
- zbentley 3y agoAdditionally, you can't easily increase/decrease consumer counts such that all consumers quickly get assigned roughly equivalent workloads. It will be interesting to watch progress on KIP-932 as the Kafka community thinks about adding message queue behavior: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...