4 ms·

I’m not sure how anyone can hate Kafka? It does what it says on the tin - move data from A to B with publish/subscribe semantics. It’s quite easy to just use

I’m not sure how anyone can hate Kafka? It does what it says on the tin - move data from A to B with publish/subscribe semantics. It’s quite easy to just use it as a dumb message broker with no retention if that’s all you need but if you do want to do something funky with persistence then go down that route. I’m not sure how anyone could have a negative feeling towards a vanilla, but rock solid and wildly popular open source tool. If they do then it will be about some niche feature or use case. I actually think this message reflects badly on the vendor here. Criticise or compete with Kafka on its technical merits if you like, but this is just a misrepresentation of their position in the market.

richieartoul 3y ago
(WarpStream co-founder) I'm not sure what you mean. Message persistence is a fundamental feature of Kafka that almost everyone using it relies on, its not some esoteric feature no one uses. We're each coming from our own network bias here, but in my experience a lot of people are really unhappy with the operational toil associated with running Kafka at scale in production.
- benjaminwootton 3y ago
  Yes it’s fundamental, but it’s not generally that significant to users of Kafka. As a developer, Kafka is a place to publish and subscribe to data with reliability and performance. As a developer, the fact that messages are persistent is nothing more than a cool feature in that I can replay messages if I need to. Things like consumer groups and offsets are features of the API, but they aren’t complex. Every similar tool whether it be RabbitMQ or IBM MQ has its own API abstractions and features. Likewise, I need to learn about failover semantics, but that’s the same with any API dependency. It seems that you and the other posters here have a concensus that it’s hard to operate. Rather than saying that Kafka is dead or a polarising technology, a better line of argument is that it’s simply hard or expensive to operate at scale. (I personally think that’s par for the course with a technology like this, but that’s an aside.) You have to remember that for everyone operating Kafka, there will be on average tens or hundreds of developers using it. And the vast, vast majority of those will not find it to be particularly polarising. Instead, they’ll find it a de-facto choice.
deanCommie 3y ago
Easy: I hate how complex it is, compared to something much simpler like https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...
mdaniel 3y ago
> I’m not sure how anyone can hate Kafka? My experience has been that such a question has two implied audiences in it: those who consume Kafka and those who have to keep the PoS alive and healthy The whole ambiguity around whether ZK is really still needed or not <https://kafka.apache.org/documentation/#zk https://kafka.apache.org/documentation/#zk> makes keeping two distributed systems alive and healthy, but don't worry you can't move your production clusters off of it anyway <https://kafka.apache.org/documentation/#kraft_zk_migration https://kafka.apache.org/documentation/#kraft_zk_migration>. It's a mess
- EdwardDiego 3y ago
  ZooKeeper is rock solid. Moving off it is a mistake, IMO. My tinfoil hat theory is that the whole impetus for KRaft is Confluent Cloud's multi-tenanted clusters have so many partitions that it starts to exceed ZK's capacities, so Confluent have built KRaft for Confluent. And yeah, the migration approach is nutso. Also very annoying, the KRaft metadata topics being changed to be super-secret for... ...some good reason, I'm sure. But it entirely removes the ability to respond to changed cluster metadata that you have with ZK, where you can watch znodes. I'm not at all a fan tbh.
  - jitl 3y ago
    > ZooKeeper is rock solid lol. lmao. never falls over or causes incidents, in the same way c is rock solid and never SIGSEGV or causes security problems
    - mdaniel 3y ago
      Some people say "well, it could be worse" but my go-to version is "at least it's not fucking etcd"
    - reissbaker 3y ago
      This thread is an excellent example of the author's point: Kafka is polarizing. Personally, in my experience with Kafka and Zookeeper at Airbnb back in the day (we also used ZK for general-purpose service discovery), they both were... temperamental. They'd chug along just fine for a bit, seemingly handling outages that e.g. RDS would have thrown a fit over, and then suddenly they'd be cataclysmically down in extremely complicated ways and be very difficult to bring back up. Even just using them required teaching a more complex mental model than most cloud-hosted offerings of similar things, and you ended up in this path dependency trap of "we already invested so much in Kafka, so if you want to send a message, use Kafka" when for like 95+% of use cases something easy like SQS would've been fine and simpler. TBQH I don't think either Kafka or ZK ever quite paid back their operational overhead cost, and personally I wouldn't recommend using either unless you absolutely need to. Warpstream looks really cool in that light!
twelve40 3y ago
i work at a place that has a whole devops department maintaining it (so not even my pain), and i still hate how overengineered the overall system ended up, it's more busywork programming for it, debugging it, etc. No reason not to use sqs or rabbit or whatever unless you have a very special use case, or begin to hit X messages/second. Or just like to spend lots of time writing boilerplate and configuring.

richieartoul 3y ago

(WarpStream co-founder) I'm not sure what you mean. Message persistence is a fundamental feature of Kafka that almost everyone using it relies on, its not some esoteric feature no one uses. We're each coming from our own network bias here, but in my experience a lot of people are really unhappy with the operational toil associated with running Kafka at scale in production.

benjaminwootton 3y ago
Yes it’s fundamental, but it’s not generally that significant to users of Kafka. As a developer, Kafka is a place to publish and subscribe to data with reliability and performance. As a developer, the fact that messages are persistent is nothing more than a cool feature in that I can replay messages if I need to. Things like consumer groups and offsets are features of the API, but they aren’t complex. Every similar tool whether it be RabbitMQ or IBM MQ has its own API abstractions and features. Likewise, I need to learn about failover semantics, but that’s the same with any API dependency. It seems that you and the other posters here have a concensus that it’s hard to operate. Rather than saying that Kafka is dead or a polarising technology, a better line of argument is that it’s simply hard or expensive to operate at scale. (I personally think that’s par for the course with a technology like this, but that’s an aside.) You have to remember that for everyone operating Kafka, there will be on average tens or hundreds of developers using it. And the vast, vast majority of those will not find it to be particularly polarising. Instead, they’ll find it a de-facto choice.

deanCommie 3y ago

Easy: I hate how complex it is, compared to something much simpler like https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...

mdaniel 3y ago

> I’m not sure how anyone can hate Kafka? My experience has been that such a question has two implied audiences in it: those who consume Kafka and those who have to keep the PoS alive and healthy The whole ambiguity around whether ZK is really still needed or not <https://kafka.apache.org/documentation/#zk https://kafka.apache.org/documentation/#zk> makes keeping two distributed systems alive and healthy, but don't worry you can't move your production clusters off of it anyway <https://kafka.apache.org/documentation/#kraft_zk_migration https://kafka.apache.org/documentation/#kraft_zk_migration>. It's a mess

EdwardDiego 3y ago
ZooKeeper is rock solid. Moving off it is a mistake, IMO. My tinfoil hat theory is that the whole impetus for KRaft is Confluent Cloud's multi-tenanted clusters have so many partitions that it starts to exceed ZK's capacities, so Confluent have built KRaft for Confluent. And yeah, the migration approach is nutso. Also very annoying, the KRaft metadata topics being changed to be super-secret for... ...some good reason, I'm sure. But it entirely removes the ability to respond to changed cluster metadata that you have with ZK, where you can watch znodes. I'm not at all a fan tbh.
- jitl 3y ago
  > ZooKeeper is rock solid lol. lmao. never falls over or causes incidents, in the same way c is rock solid and never SIGSEGV or causes security problems
  - mdaniel 3y ago
    Some people say "well, it could be worse" but my go-to version is "at least it's not fucking etcd"
  - reissbaker 3y ago
    This thread is an excellent example of the author's point: Kafka is polarizing. Personally, in my experience with Kafka and Zookeeper at Airbnb back in the day (we also used ZK for general-purpose service discovery), they both were... temperamental. They'd chug along just fine for a bit, seemingly handling outages that e.g. RDS would have thrown a fit over, and then suddenly they'd be cataclysmically down in extremely complicated ways and be very difficult to bring back up. Even just using them required teaching a more complex mental model than most cloud-hosted offerings of similar things, and you ended up in this path dependency trap of "we already invested so much in Kafka, so if you want to send a message, use Kafka" when for like 95+% of use cases something easy like SQS would've been fine and simpler. TBQH I don't think either Kafka or ZK ever quite paid back their operational overhead cost, and personally I wouldn't recommend using either unless you absolutely need to. Warpstream looks really cool in that light!

twelve40 3y ago

i work at a place that has a whole devops department maintaining it (so not even my pain), and i still hate how overengineered the overall system ended up, it's more busywork programming for it, debugging it, etc. No reason not to use sqs or rabbit or whatever unless you have a very special use case, or begin to hit X messages/second. Or just like to spend lots of time writing boilerplate and configuring.