5 ms·
Building Self-driving Kafka clusters using open source components
- pram 4y agoTheir upgrade strategy is really awful. Kafka can do rolling upgrades, it's one of the few things that has worked reliably well for me in Kafka over the past 6 tears. Pretty much as simple as a broker reboot and setting inter.broker.protocol.version/log.message.format.version correctly. They made it way harder than it needs to be.
- seanc 4y agoI'm confused about the breezy way in which they distribute partitions and consumers. Since the producer computes the partition key and the consumer chooses which partitions to bind, how does the central kafka team enforce balance? One would need some sort of policy about how developers use the Producer and Consumer API's for these things, but that would be a very difficult thing to manage and enforce I would think.
- Hackbraten 4y agoI hate Kafka with a passion. We’re driving it with Kafka Streams, and our Streams threads have been blowing up left and right with undecipherable error messages whenever we look at it slightly wrong. We also don’t have the resources to babysit it hourly so here we are, plagued with undebuggable production downtime and no way out of this. I’ve been writing software for a living for more than two decades, and Kafka is the most horrible, impossible-to-grasp boat anchor I’ve ever had to deal with. I hate every minute of my working day that I have to deal with Kafka Streams’s hissy fits.
- codegangsta 4y agoHave you tried nats.io and it’s new streaming tech, Jetstream? Night and day difference in terms of management and ease of use. In my last org I was fighting so hard against folks wanting to adopt Kafka, and then I found out about nats, total game changer
- igetspam 4y agoOr Redpanda. We like to think it's Kafka but better. Obvious bias: I work for Redpanda
- SushiMon 4y agoNATS is open source and CNCF. Redpanda is more a SaaS -- if it goes away, so does your app.
- FridgeSeal 4y agoI don’t think that holds here, Redpanda has a Kafka compatible API layer and you can 100% self-host it. Being CNCF or not doesn’t seem like a meaningful or useful distinction here.
- EdwardDiego 4y agoThe compatibility isn't 100% there yet.
- igetspam 4y agohttps://github.com/redpanda-data/redpanda https://github.com/redpanda-data/redpanda You're welcome to use it yourself and not pay us a dime. We're a business so we need to monetize but our core is to build a better Kafka. The nice thing is that if you've already invested in building against Kafka but you're exhausted by running Kafka, ours works better and faster. (Again, I'm biased so don't trust me, verify.)
- FridgeSeal 4y agoNATS and Kafka offer different semantics. By default, NATS offers non durable messages and different delivery options. I believe you could get durable messages, but last time I read up on this, it required running extra features or infrastructure to support this.
- codegangsta 4y ago
- EdwardDiego 4y agoYou're conflating Apache Kafka, the distributed log and Kafka Streams, the streaming library built by Confluent that leverages Apache Kafka. Kafka itself is straightforward to grasp, Kafka Streams, yep, your concerns sound familiar.
- d_t_w 4y agoKafka Streams is a core part of the open-source Apache Kafka project. You may be mistaking it for kSQL, or ksqlDB as they call it now, which is a Confluent project built on top of Kafka Streams.
- seanc 4y agoBe that as it may, I agree with the spirit of the grand-parent comment, which is that if you stick to straightforward Pub/Sub Kafka works great.
- EdwardDiego 4y agoIt is, but it's ancillary to Kafka the distributed log.
- mudrockbestgirl 4y agoSame experience here. We're using Kafka Streams and it's a horrible experience due to its unreliability. We need a full-time person just to make sure it stays up.
- geodel 4y agoSame experience here. The only thing excellent about Kafka is marketing. It is endless, relentless and everywhere.