5 ms·

I don't think this is coming up enough in the comments! Kafka is a gigantic pain in the ass to run. Configuration is torture. I haven't used Pulsar, but please

by cmckn 6y ago

I don't think this is coming up enough in the comments! Kafka is a gigantic pain in the ass to run. Configuration is torture. I haven't used Pulsar, but please if someone has a message broker that's easy to run, let me know.

z9e 6y ago
Like most Apache products it comes with a lot of knobs and switches, the more you tweak those the worse it gets. Kafka runs completely fine as long as you aren’t trying to tune every config option, unless you have a super custom use case and you know what you’re doing.
- cmckn 6y ago
  I had trouble accomplishing simple things like setting the maximum message size. You have to: 1. Either increase the default max message size on the broker, or set it on the specific topic(s) involved. 2. Set the max message size at every producer. 3. Set the max batch size at every consumer, which has some interaction with a message count (if I'm remembering correctly). Because we increased the allowed message size, I wanted to make sure that disk space was freed after a given interval. This was super opaque to get working. I don't remember the specific config keys, but there was about 4 parameters involved, and 3 had no effect if the 4th wasn't set properly. They all used sligtly different naming conventions, so didn't appear in context in the documentation. Still, I agree that defaults are usually the way to go with Apache stuff. If you find yourself turning a lot of knobs to stabilize something, there's probably a design issue.
manigandham 6y ago
Pulsar currently is easy to setup if you're on Kubernetes, especially if you can dedicate an entire cluster to it. You can run a small 3-node cluster on GKE with local SSD instance storage and GCS/S3 tiering that can handle GBit throughput. Otherwise look at NATS with NATS Streaming (although they're currently redesigning a better version of it). There's also Liftbridge which is an alternative NATS Streaming fork. https://docs.nats.io/ https://docs.nats.io/ https://docs.nats.io/nats-streaming-concepts/intro https://docs.nats.io/nats-streaming-concepts/intro https://liftbridge.io/ https://liftbridge.io/

by cmckn 6y ago

z9e 6y ago

Like most Apache products it comes with a lot of knobs and switches, the more you tweak those the worse it gets. Kafka runs completely fine as long as you aren’t trying to tune every config option, unless you have a super custom use case and you know what you’re doing.

cmckn 6y ago
I had trouble accomplishing simple things like setting the maximum message size. You have to: 1. Either increase the default max message size on the broker, or set it on the specific topic(s) involved. 2. Set the max message size at every producer. 3. Set the max batch size at every consumer, which has some interaction with a message count (if I'm remembering correctly). Because we increased the allowed message size, I wanted to make sure that disk space was freed after a given interval. This was super opaque to get working. I don't remember the specific config keys, but there was about 4 parameters involved, and 3 had no effect if the 4th wasn't set properly. They all used sligtly different naming conventions, so didn't appear in context in the documentation. Still, I agree that defaults are usually the way to go with Apache stuff. If you find yourself turning a lot of knobs to stabilize something, there's probably a design issue.

manigandham 6y ago

Pulsar currently is easy to setup if you're on Kubernetes, especially if you can dedicate an entire cluster to it. You can run a small 3-node cluster on GKE with local SSD instance storage and GCS/S3 tiering that can handle GBit throughput. Otherwise look at NATS with NATS Streaming (although they're currently redesigning a better version of it). There's also Liftbridge which is an alternative NATS Streaming fork. https://docs.nats.io/ https://docs.nats.io/ https://docs.nats.io/nats-streaming-concepts/intro https://docs.nats.io/nats-streaming-concepts/intro https://liftbridge.io/ https://liftbridge.io/