4 ms·
exactly once messaging is a lie. it just mean you have duplicates message but let Kafka try to detect duplicate for you, which is almost never what you want. it
by skyde 6y ago
exactly once messaging is a lie. it just mean you have duplicates message but let Kafka try to detect duplicate for you, which is almost never what you want. it’s much simpler to make your message processing indempotent.
- opportune 6y agoGreat comment. I have seen multiple people in my short career get burned by trying to figure out how to get exactly once semantics in a distributed system. The solution almost always, as you said, is idempotency (perhaps with the message producer appending a UUID field).
- halbritt 6y agoKafka has exactly-once semantics that mostly work. They also have "idempotent producer" which ostensibly provides atomic transactions as well. Sounds awesome, but the implementation pretty much breaks kafka in that one can no longer scale. Topic and partition are all constrained to a single broker. Hot spots abound!