3 ms·

Is there a good, simple alternative to Kafka? My experience so far with Kafka is that error messages are nearly useless. Even an invalid command line parameter

by blagie 4y ago

Is there a good, simple alternative to Kafka? My experience so far with Kafka is that error messages are nearly useless. Even an invalid command line parameter raises an exception, rather than printing a usage. In many cases, if I do something wrong, it just stalls and times out, with no real way to debug. The documentation caps out at a very nominal level. I tried to implement a simple KVS with compact logs. Documentation ran out and I was wading in deep waters. I'm trying to build a platform that's developer-friendly, and I'm deep enough in to know Kafka shouldn't be part of it. That's not to mention the growing uncertainty around the JDK/JVM. I want open-source, archival, and to be able to reply event streams from the beginning. For my application, I'll take lower performance in return for better usability.

EdwardDiego 4y ago
NATS maybe? https://github.com/nats-io https://github.com/nats-io Yeah, Kafka's complex, and it's really a tool that you need to think long and hard about whether or not your problems are complicated enough that introducing Kafka to solve them will ultimately reduce complexity. And it's definitely something that developers have to learn the semantics of to use well. > The documentation caps out at a very nominal level. I tried to implement a simple KVS with compact logs. Documentation ran out and I was wading in deep waters I occasionally contribute to Apache Kafka, any pointers on what documentation was lacking, and where you could have used more? I will see if I can contribute something around that :) > That's not to mention the growing uncertainty around the JDK/JVM. What do you mean by this? > I want open-source, archival, and to be able to reply event streams from the beginning. Yeah, maybe check out Liftbridge, it's built on NATS. https://liftbridge.io/ https://liftbridge.io/ Apache Pulsar might be worth a look, but it's actually more complex under the hood than Kafka, but has a lot of features built-in that either aren't in FOSS Kafka yet, like tiered storage, or won't be until Confluent doesn't dominate the PMC (like an integrated schema registry), or just can't be done very nicely, if at all, like decent multi-tenancy. That said, it's a fast moving target, the code quality last I looked was patchy in places, ditto the documentation for both it and Bookkeeper, and the admin overhead is higher (managing bookies and brokers and Zookeepers vs. just brokers and ZK with Kafka, or when KRaft is production ready, just brokers).
- jamesfinlayson 4y ago
  >> That's not to mention the growing uncertainty around the JDK/JVM. > What do you mean by this? Agreed - four years ago Oracle changed all the licensing but I thought they did a bit of a 180 on that in the last year or two.
  - EdwardDiego 4y ago
    Yep, agree. I thought that they might have been referring to that recent log4j2 vuln, but yeah, clarification welcomed.
- blagie 4y ago
  Thank you! > What do you mean by this? Oracle. Oracle is trying to milk the cash cow. That's and unpredictable ride. > I occasionally contribute to Apache Kafka, any pointers on what documentation was lacking, and where you could have used more? I will see if I can contribute something around that :) Well, if you're contributing, there are three Python interfaces to Kafka: confluent, pykafka, and python-kafka: - All three have major gaps. - None use async Rallying around a single, usable interface would be a huge step. That's where I'd most encourage contribution. But to see gaps in documentation, take any of the three, and try to fetch the most recent event (without blocking). You'll run into a bunch of holes along the way. I wanted a simple KVS in Kafka -- store with compact, and fetch most recent event. That would allow the system to run without a DB. That was not trivial to figure out.
  - pixelmonkey 4y ago
    I just wanted to mention that pykafka is currently unmaintained and archived on GitHub: https://github.com/Parsely/pykafka https://github.com/Parsely/pykafka pykafka was originally developed and maintained by my team at Parse.ly, but we no longer maintain it. We instead encourage folks to use confluent-kafka-python, which is what we have ourselves switched to in our production systems: https://github.com/confluentinc/confluent-kafka-python https://github.com/confluentinc/confluent-kafka-python (pykafka was developed at a time before Confluent invested in their own Python binding. Some of the history of the project is described in this 2016 blog post[1] and our original 2015 announcement[2].) [1]: https://blog.parse.ly/pykafka-now/ https://blog.parse.ly/pykafka-now/ [2]: https://blog.parse.ly/announcing-pykafka-python-support-for-kafka-0-8-x/ https://blog.parse.ly/announcing-pykafka-python-support-for-...

by blagie 4y ago

EdwardDiego 4y ago

NATS maybe? https://github.com/nats-io https://github.com/nats-io Yeah, Kafka's complex, and it's really a tool that you need to think long and hard about whether or not your problems are complicated enough that introducing Kafka to solve them will ultimately reduce complexity. And it's definitely something that developers have to learn the semantics of to use well. > The documentation caps out at a very nominal level. I tried to implement a simple KVS with compact logs. Documentation ran out and I was wading in deep waters I occasionally contribute to Apache Kafka, any pointers on what documentation was lacking, and where you could have used more? I will see if I can contribute something around that :) > That's not to mention the growing uncertainty around the JDK/JVM. What do you mean by this? > I want open-source, archival, and to be able to reply event streams from the beginning. Yeah, maybe check out Liftbridge, it's built on NATS. https://liftbridge.io/ https://liftbridge.io/ Apache Pulsar might be worth a look, but it's actually more complex under the hood than Kafka, but has a lot of features built-in that either aren't in FOSS Kafka yet, like tiered storage, or won't be until Confluent doesn't dominate the PMC (like an integrated schema registry), or just can't be done very nicely, if at all, like decent multi-tenancy. That said, it's a fast moving target, the code quality last I looked was patchy in places, ditto the documentation for both it and Bookkeeper, and the admin overhead is higher (managing bookies and brokers and Zookeepers vs. just brokers and ZK with Kafka, or when KRaft is production ready, just brokers).

jamesfinlayson 4y ago
>> That's not to mention the growing uncertainty around the JDK/JVM. > What do you mean by this? Agreed - four years ago Oracle changed all the licensing but I thought they did a bit of a 180 on that in the last year or two.
- EdwardDiego 4y ago
  Yep, agree. I thought that they might have been referring to that recent log4j2 vuln, but yeah, clarification welcomed.
blagie 4y ago
Thank you! > What do you mean by this? Oracle. Oracle is trying to milk the cash cow. That's and unpredictable ride. > I occasionally contribute to Apache Kafka, any pointers on what documentation was lacking, and where you could have used more? I will see if I can contribute something around that :) Well, if you're contributing, there are three Python interfaces to Kafka: confluent, pykafka, and python-kafka: - All three have major gaps. - None use async Rallying around a single, usable interface would be a huge step. That's where I'd most encourage contribution. But to see gaps in documentation, take any of the three, and try to fetch the most recent event (without blocking). You'll run into a bunch of holes along the way. I wanted a simple KVS in Kafka -- store with compact, and fetch most recent event. That would allow the system to run without a DB. That was not trivial to figure out.
- pixelmonkey 4y ago
  I just wanted to mention that pykafka is currently unmaintained and archived on GitHub: https://github.com/Parsely/pykafka https://github.com/Parsely/pykafka pykafka was originally developed and maintained by my team at Parse.ly, but we no longer maintain it. We instead encourage folks to use confluent-kafka-python, which is what we have ourselves switched to in our production systems: https://github.com/confluentinc/confluent-kafka-python https://github.com/confluentinc/confluent-kafka-python (pykafka was developed at a time before Confluent invested in their own Python binding. Some of the history of the project is described in this 2016 blog post[1] and our original 2015 announcement[2].) [1]: https://blog.parse.ly/pykafka-now/ https://blog.parse.ly/pykafka-now/ [2]: https://blog.parse.ly/announcing-pykafka-python-support-for-kafka-0-8-x/ https://blog.parse.ly/announcing-pykafka-python-support-for-...