3 ms·
The kafka protocol is a distributed write ahead log. If you want a job queue you need to build something on top of that, it’s a pretty low level primative.
by rockwotj 2y ago
The kafka protocol is a distributed write ahead log. If you want a job queue you need to build something on top of that, it’s a pretty low level primative.
- atmosx 2y agoWhy does everybody keep missing this point? I don’t know.
- nobleach 2y agoThere's a wonderful Kafka Children's book that I always suggest every team I work with read: https://www.gentlydownthe.stream/ https://www.gentlydownthe.stream/ The way I describe Kafka is, "an event has transpired... sometimes you care, and choose to take an action based on that event" The way I describe RabbitMQ is, "there's a new ticket in the lineup... it needs to be grabbed for action or left in the lineup... or discarded" Definitely not perfect analogies. But they get the point across that Kafka is designed to be reactive and message queues/job queues are meant to be more imperative.
- stickfigure 2y agoYour two-sentence description is excellent. That book, not so much.
- nobleach 2y agoI suppose that's fair.
- mumrah 2y agoNot for long. An early access version of KIP-932 Queues for Kafka will be released in 4.0 in a few weeks. https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...