4 ms·
I hate Kafka with a passion. We’re driving it with Kafka Streams, and our Streams threads have been blowing up left and right with undecipherable error messages
by Hackbraten 4y ago
I hate Kafka with a passion. We’re driving it with Kafka Streams, and our Streams threads have been blowing up left and right with undecipherable error messages whenever we look at it slightly wrong. We also don’t have the resources to babysit it hourly so here we are, plagued with undebuggable production downtime and no way out of this.
I’ve been writing software for a living for more than two decades, and Kafka is the most horrible, impossible-to-grasp boat anchor I’ve ever had to deal with.
I hate every minute of my working day that I have to deal with Kafka Streams’s hissy fits.
- codegangsta 4y agoHave you tried nats.io and it’s new streaming tech, Jetstream? Night and day difference in terms of management and ease of use. In my last org I was fighting so hard against folks wanting to adopt Kafka, and then I found out about nats, total game changer
- igetspam 4y agoOr Redpanda. We like to think it's Kafka but better. Obvious bias: I work for Redpanda
- SushiMon 4y agoNATS is open source and CNCF. Redpanda is more a SaaS -- if it goes away, so does your app.
- FridgeSeal 4y agoI don’t think that holds here, Redpanda has a Kafka compatible API layer and you can 100% self-host it. Being CNCF or not doesn’t seem like a meaningful or useful distinction here.
- EdwardDiego 4y agoThe compatibility isn't 100% there yet.
- igetspam 4y agohttps://github.com/redpanda-data/redpanda https://github.com/redpanda-data/redpanda You're welcome to use it yourself and not pay us a dime. We're a business so we need to monetize but our core is to build a better Kafka. The nice thing is that if you've already invested in building against Kafka but you're exhausted by running Kafka, ours works better and faster. (Again, I'm biased so don't trust me, verify.)
- FridgeSeal 4y agoNATS and Kafka offer different semantics. By default, NATS offers non durable messages and different delivery options. I believe you could get durable messages, but last time I read up on this, it required running extra features or infrastructure to support this.
- codegangsta 4y agoYou now get durability on whatever subjects you want via NATS Jetstream: https://docs.nats.io/nats-concepts/jetstream https://docs.nats.io/nats-concepts/jetstream No additional infrastructure, just enable it on the nodes you want persistence on and create a stream. Publishers don't even have to change, consumers have a great api with lots of options
- FridgeSeal 4y agoOh that’s pretty cool, I wasn’t aware of that! Nice to that functionality added!
- EdwardDiego 4y agoYou're conflating Apache Kafka, the distributed log and Kafka Streams, the streaming library built by Confluent that leverages Apache Kafka. Kafka itself is straightforward to grasp, Kafka Streams, yep, your concerns sound familiar.
- d_t_w 4y agoKafka Streams is a core part of the open-source Apache Kafka project. You may be mistaking it for kSQL, or ksqlDB as they call it now, which is a Confluent project built on top of Kafka Streams.
- seanc 4y agoBe that as it may, I agree with the spirit of the grand-parent comment, which is that if you stick to straightforward Pub/Sub Kafka works great.
- EdwardDiego 4y agoIt is, but it's ancillary to Kafka the distributed log.
- mudrockbestgirl 4y agoSame experience here. We're using Kafka Streams and it's a horrible experience due to its unreliability. We need a full-time person just to make sure it stays up.
- geodel 4y agoSame experience here. The only thing excellent about Kafka is marketing. It is endless, relentless and everywhere.