7 ms·

I'm surprised by the amount of criticism in this thread. I've used Kafka in the past and it definitely got the job done (as a message bus, not using stream proc

I'm surprised by the amount of criticism in this thread. I've used Kafka in the past and it definitely got the job done (as a message bus, not using stream processing or the other more whiz-bang features). What do people use instead?

skyde 7y ago
If you need. Ordered log abstraction (Apache pulsar, Facebook log device ...) if you need a real transactional and highly available message broker any (my/jms) server like Rabbitmq IBM MQ .... Kafka in my experience is always the worse solution unless you need to aggregate http server log to do offline analytics using something like spark
halbritt 7y ago
For streaming data kafka is great. It gets misused in many ways. It can be made to guarantee order and has semantics for exactly once processing which may lead one to believe it can be used for transactions, but that forces all kinds of things to occur under the hood that makes performance at scale nearly impossible. I know folks like Netflix are using it for streaming UI logs into Flink for real time sentiment analysis. There are some very serious use cases for Kafka at scale. If one just needs a message broker, well, there are other, better ones.
- skyde 7y ago
  exactly once messaging is a lie. it just mean you have duplicates message but let Kafka try to detect duplicate for you, which is almost never what you want. it’s much simpler to make your message processing indempotent.
  - opportune 7y ago
    Great comment. I have seen multiple people in my short career get burned by trying to figure out how to get exactly once semantics in a distributed system. The solution almost always, as you said, is idempotency (perhaps with the message producer appending a UUID field).
    - halbritt 7y ago
      Kafka has exactly-once semantics that mostly work. They also have "idempotent producer" which ostensibly provides atomic transactions as well. Sounds awesome, but the implementation pretty much breaks kafka in that one can no longer scale. Topic and partition are all constrained to a single broker. Hot spots abound!
- piyh 7y ago
  My company uses it for batch file processing. The CIO demanded we use it instead of something like FTP, but didn't understand the software patterns it was built for. We now have a very overly complex file transmission system and a new CIO.
skyde 7y ago
my experience is the opposite unless you really need the whiz-bang feature you should never use Kafka it’s the least reliable and hardest to run it require active babysitting by skilled team of admin and ops. if you don’t need low end-to-end latency (tailing consumer blocked on long polling) it’s better to use something like HdFS or AWS S3, but if you need it but have low throughout it’s better to use something like RabbitMQ. if you have both high throughout and need polling then it’s worth investing into Apache Pulsar.
- maxdo 7y ago
  Pulsar is half baked with limited number of half baked clients even for major languages. Kafka instead is a bulletproof solution with tons of clients in any flavor, with instruments to detect issues, alert about them. Don't even compare them.
  - skyde 7y ago
    Pulsar itself is solid but it’s true number of good clients is limited. Same is true for kafka the only reliable client is the java one. All other are based on librdkafka which still have many major bugs causing it to be stuck and not producing to new leader during leader changes. One benefit of pulsar is that the client library is completely stateless and much simple. It does not need to know where data is stored it’s all handled by the pulsar broker.
  - mianos 7y ago
    I have used the C++, python and Java clients and they are all pretty usable. Pulsar deployment support is great. It seems the teams deploy it themselves.
  - rocho 6y ago
    I just checked and they have a websocket API which I imagine is usable in almost every language. We might use it ourselves instead of the official Python library which is not async. Async publishing and consuming is a requirement for our project.
  - manigandham 6y ago
whatsmyusername 7y ago
AWS SQS, RabbitMQ, or Redis. I would build on one of those 3 first, then migrate when throughput became an issue. I've had good experiences with Kinesis Firehose as well. I've yet to run into a workload that actually needed Kafka (and if you're not a Java shop don't add Java pieces unless you know you need what those Java pieces provide).

skyde 7y ago

If you need. Ordered log abstraction (Apache pulsar, Facebook log device ...) if you need a real transactional and highly available message broker any (my/jms) server like Rabbitmq IBM MQ .... Kafka in my experience is always the worse solution unless you need to aggregate http server log to do offline analytics using something like spark

halbritt 7y ago

For streaming data kafka is great. It gets misused in many ways. It can be made to guarantee order and has semantics for exactly once processing which may lead one to believe it can be used for transactions, but that forces all kinds of things to occur under the hood that makes performance at scale nearly impossible. I know folks like Netflix are using it for streaming UI logs into Flink for real time sentiment analysis. There are some very serious use cases for Kafka at scale. If one just needs a message broker, well, there are other, better ones.

skyde 7y ago
exactly once messaging is a lie. it just mean you have duplicates message but let Kafka try to detect duplicate for you, which is almost never what you want. it’s much simpler to make your message processing indempotent.
- opportune 7y ago
  Great comment. I have seen multiple people in my short career get burned by trying to figure out how to get exactly once semantics in a distributed system. The solution almost always, as you said, is idempotency (perhaps with the message producer appending a UUID field).
  - halbritt 7y ago
    Kafka has exactly-once semantics that mostly work. They also have "idempotent producer" which ostensibly provides atomic transactions as well. Sounds awesome, but the implementation pretty much breaks kafka in that one can no longer scale. Topic and partition are all constrained to a single broker. Hot spots abound!
piyh 7y ago
My company uses it for batch file processing. The CIO demanded we use it instead of something like FTP, but didn't understand the software patterns it was built for. We now have a very overly complex file transmission system and a new CIO.

skyde 7y ago

my experience is the opposite unless you really need the whiz-bang feature you should never use Kafka it’s the least reliable and hardest to run it require active babysitting by skilled team of admin and ops. if you don’t need low end-to-end latency (tailing consumer blocked on long polling) it’s better to use something like HdFS or AWS S3, but if you need it but have low throughout it’s better to use something like RabbitMQ. if you have both high throughout and need polling then it’s worth investing into Apache Pulsar.

maxdo 7y ago
Pulsar is half baked with limited number of half baked clients even for major languages. Kafka instead is a bulletproof solution with tons of clients in any flavor, with instruments to detect issues, alert about them. Don't even compare them.
- skyde 7y ago
  Pulsar itself is solid but it’s true number of good clients is limited. Same is true for kafka the only reliable client is the java one. All other are based on librdkafka which still have many major bugs causing it to be stuck and not producing to new leader during leader changes. One benefit of pulsar is that the client library is completely stateless and much simple. It does not need to know where data is stored it’s all handled by the pulsar broker.
- mianos 7y ago
  I have used the C++, python and Java clients and they are all pretty usable. Pulsar deployment support is great. It seems the teams deploy it themselves.
- rocho 6y ago
  I just checked and they have a websocket API which I imagine is usable in almost every language. We might use it ourselves instead of the official Python library which is not async. Async publishing and consuming is a requirement for our project.
- manigandham 6y ago

whatsmyusername 7y ago

AWS SQS, RabbitMQ, or Redis. I would build on one of those 3 first, then migrate when throughput became an issue. I've had good experiences with Kinesis Firehose as well. I've yet to run into a workload that actually needed Kafka (and if you're not a Java shop don't add Java pieces unless you know you need what those Java pieces provide).