3 ms·
For streaming data kafka is great. It gets misused in many ways. It can be made to guarantee order and has semantics for exactly once processing which may lea
by halbritt 6y ago
For streaming data kafka is great. It gets misused in many ways. It can be made to guarantee order and has semantics for exactly once processing which may lead one to believe it can be used for transactions, but that forces all kinds of things to occur under the hood that makes performance at scale nearly impossible.
I know folks like Netflix are using it for streaming UI logs into Flink for real time sentiment analysis. There are some very serious use cases for Kafka at scale.
If one just needs a message broker, well, there are other, better ones.
- skyde 6y agoexactly once messaging is a lie. it just mean you have duplicates message but let Kafka try to detect duplicate for you, which is almost never what you want. it’s much simpler to make your message processing indempotent.
- opportune 6y agoGreat comment. I have seen multiple people in my short career get burned by trying to figure out how to get exactly once semantics in a distributed system. The solution almost always, as you said, is idempotency (perhaps with the message producer appending a UUID field).
- halbritt 6y agoKafka has exactly-once semantics that mostly work. They also have "idempotent producer" which ostensibly provides atomic transactions as well. Sounds awesome, but the implementation pretty much breaks kafka in that one can no longer scale. Topic and partition are all constrained to a single broker. Hot spots abound!
- piyh 6y agoMy company uses it for batch file processing. The CIO demanded we use it instead of something like FTP, but didn't understand the software patterns it was built for. We now have a very overly complex file transmission system and a new CIO.