4 ms·
This seems very much like saying that if you shoot the hostage in a hostage situation, you no longer have a hostage situation. Like...the whole point of having
by XorNot 6y ago
This seems very much like saying that if you shoot the hostage in a hostage situation, you no longer have a hostage situation.
Like...the whole point of having messages work that way is that you're not having to bake event processing and duplicate transaction handling and compensating transactions and all the other message handling logic into literally every process which comes after your message queue.
It's a non-solution - you're just leaking the problem out to another part of the stack.
- Joeri 6y agoIt isn’t actually that complicated. The kafka client abstracts away the complexity of tracking offsets. You fetch batches of messages through the kafka client in a loop, process them one by one, and commit the batch’s offset if everything went well. If something goed wrong, after restarting the consumer the kafka client will make sure it gets offered the same batch again. So you’re in an at least once scenario where as long as you process messages in an idempotent way and skip the bad ones it is basically foolproof.
- rad_gruchalski 6y agoIndeed. Plus it automatically handles consumer group rebelancing and partition leader changes. From the user perspective, one only needs to ensure having enough consumers in a consumer group. At Klarrio, we have a rather small cluster in production but we can easily push 2.4Gbps with <100ms end to end latency for dozens of consumers. It’s working really well.