3 ms·

I hear from everyone using Kafka in production that it is hell unless you use Confluent. I gave a try to NATS JetStreams but I havn't been convinced by the per

I hear from everyone using Kafka in production that it is hell unless you use Confluent. I gave a try to NATS JetStreams but I havn't been convinced by the performances of the Python client, nor the JavaScript one. I don't have extreme data, I just need descent performances. I'm thinking about giving a try to RabbitMQ streams. I have been very happy with RabbitMQ, the MQTT plugin isn't fully working (the big one is that retained messages are not sent to wildcard subscribers), but it should work with AMQP.

EdwardDiego 4y ago
Welp, here's a dissenting opinion - it's not. I've run self-managed, sorta managed (MSK), fully managed (Confluent Cloud), and somewhat managed (Strimzi). It is complex, yes, but it solves a very complicated problem. The issue tends to arise when people use it when simpler alternatives exist for their problem.
amath 4y ago
Have you tried Pulsar or Redpanda? Both seem mature enough and provide decent performance to probably meet your needs. What I hear is that Redpanda is a lot easier to manage than Kafka.
- speedgoose 4y ago
  I haven’t took the time but I should. Redpanda is written in C++ and I tend to prefer safer programming languages, redis being the exception.
- serial_dev 4y ago
  What's the license for Redpanda? I couldn't find anything but maybe it's because I'm on mobile.
  - deniscoady 4y ago
    Disclaimer, I work at Redpanda. Redpanda Community Edition is licensed with the Redpanda Business Source License (BSL). The core features are free and source-available. This license was inspired by MariaDB and CockroachDB and "for 99.999% of users, restrictions will not apply". The big restriction is that users "cannot provide Redpanda as a commercial streaming or queuing service to others" which is primarily to deter large cloud vendors from taking our work and impacting our ability to operate as a business. Here are some links for more details on how we came to this decision and how we license all our products and features. https://redpanda.com/blog/bsl-source-available-license https://redpanda.com/blog/bsl-source-available-license https://docs.redpanda.com/docs/get-started/licenses/ https://docs.redpanda.com/docs/get-started/licenses/
mikedelago 4y ago
Kafka (along with zookeeper) really isn't that bad to self-host. Ime, it's easy if the org has a half decent infrastructure/configuration as code setup.
grepLeigh 4y ago
I've had an excellent experience using the Rust NATS client! I pump time series data through NATS running on a Raspberry Pi, which is part of a 3D printer monitoring and event/automation system. I also use NATS as an MQTT broker, for compatibility with other software in the 3D printer ecosystem. FWIW I also have lots of experience running large Kafka and Rabbitmq fleets. The choice between these technologies depends on what you're optimizing for.
outworlder 4y ago
> I'm thinking about giving a try to RabbitMQ We went the opposite route. Kafka has been much better. Up to a certain volume, both solved the problem. When RabbitMQ required too much tuning, a decision was made to go to Kafka, and it's been stellar. Both are pretty good, but understand that there are too many variables involved and you can't really escape production hell indefinitely, regardless of what you pick. What changes is when you are going to see the flames, and what is going to spark them.

EdwardDiego 4y ago

Welp, here's a dissenting opinion - it's not. I've run self-managed, sorta managed (MSK), fully managed (Confluent Cloud), and somewhat managed (Strimzi). It is complex, yes, but it solves a very complicated problem. The issue tends to arise when people use it when simpler alternatives exist for their problem.

amath 4y ago

Have you tried Pulsar or Redpanda? Both seem mature enough and provide decent performance to probably meet your needs. What I hear is that Redpanda is a lot easier to manage than Kafka.

speedgoose 4y ago
I haven’t took the time but I should. Redpanda is written in C++ and I tend to prefer safer programming languages, redis being the exception.
serial_dev 4y ago
What's the license for Redpanda? I couldn't find anything but maybe it's because I'm on mobile.
- deniscoady 4y ago
  Disclaimer, I work at Redpanda. Redpanda Community Edition is licensed with the Redpanda Business Source License (BSL). The core features are free and source-available. This license was inspired by MariaDB and CockroachDB and "for 99.999% of users, restrictions will not apply". The big restriction is that users "cannot provide Redpanda as a commercial streaming or queuing service to others" which is primarily to deter large cloud vendors from taking our work and impacting our ability to operate as a business. Here are some links for more details on how we came to this decision and how we license all our products and features. https://redpanda.com/blog/bsl-source-available-license https://redpanda.com/blog/bsl-source-available-license https://docs.redpanda.com/docs/get-started/licenses/ https://docs.redpanda.com/docs/get-started/licenses/

mikedelago 4y ago

Kafka (along with zookeeper) really isn't that bad to self-host. Ime, it's easy if the org has a half decent infrastructure/configuration as code setup.

grepLeigh 4y ago

I've had an excellent experience using the Rust NATS client! I pump time series data through NATS running on a Raspberry Pi, which is part of a 3D printer monitoring and event/automation system. I also use NATS as an MQTT broker, for compatibility with other software in the 3D printer ecosystem. FWIW I also have lots of experience running large Kafka and Rabbitmq fleets. The choice between these technologies depends on what you're optimizing for.

outworlder 4y ago

> I'm thinking about giving a try to RabbitMQ We went the opposite route. Kafka has been much better. Up to a certain volume, both solved the problem. When RabbitMQ required too much tuning, a decision was made to go to Kafka, and it's been stellar. Both are pretty good, but understand that there are too many variables involved and you can't really escape production hell indefinitely, regardless of what you pick. What changes is when you are going to see the flames, and what is going to spark them.