3 ms·
> The Kafka-consumer process is responsible for fetching messages from Kafka, and placing them on a local queue that is read by the task-execution processes Ah
by kerblang 6y ago
> The Kafka-consumer process is responsible for fetching messages from Kafka, and placing them on a local queue that is read by the task-execution processes
Ah, yep, as usual you end up with a queue for your queue because Kafka isn't actually a queue in the first place. This is why I recommend Kafka for _data_, rabbit for _tasks_. The article doesn't really explain how this works, but I'm guessing that one service holds commands in memory, then sends http messages to the workers. As they noted, you end up with lost tasks on restart, and that's the big gaping hole that a lot of people would not want to fall into - your mileage may vary. I assume the kafka reader service has to also track how busy each worker is, also lost on restart.
- thebean11 6y agoIf you have some mapping from data to units of work, can't you just use callbacks to make sure the data isn't acked until the units of work complete successfully? I use kafka at work and so far haven't really run into this as an issue. In the bit you quoted, I'd assume the local work queue has callback objects.
- kerblang 6y agoSorry for late answer: A Kafka ack is not a signal that an individual message is complete, but that the message index pointer can move forwards. So if you read 10 tasks and run them in parallel, you can't just ack a random selection of 5 as they complete. You could work out a system to ack the entire bunch when the last one finishes though. All of this assumes you're ok with more-than-once delivery, of course.