9 ms·

I've used Kafka a lot in the past and I've been looking at Pulsar. The broker being stateless is great. I've had issues with moving partitions around in Kafka.

I've used Kafka a lot in the past and I've been looking at Pulsar. The broker being stateless is great. I've had issues with moving partitions around in Kafka. That being said, I'm getting tired of managing these clunky memory-hungry JVM-based systems that rely on external dependencies like Apache Zookeeper and BookKeeper. They may be necessary if you're Yahoo, but I would argue that for the vast majority of companies, these complex systems create so many administration headaches that the net productivity impact is negative. I've spend days or weeks debugging Kafka issues. I almost feel like "big clunky heavy JVM enterprise software" has become synonymous with Apache projects. If you don't absolutely need the messaging guarantees I would recommend looking into NATS (https://nats.io/ https://nats.io/) - it's a brokered systems that's significantly more lightweight and super easy to deploy and play around with. You can get persistence and delivery guarantees with NATS streaming, but that part is optional and a bit more early stage.

KptMarchewa 6y ago
>That being said, I'm getting tired of managing these clunky memory-hungry JVM-based systems that rely on external dependencies like Apache Zookeeper and BookKeeper. One of my pet peeves with Kafka were CLI tools written in JVM languages. Slow JVM startup time was killing my focus.
- gas9S9zw3P9c 6y ago
  Yep. I'm using containers and Kubernetes in production so I'm trying to stay far away from anything JVM-based. On top of being slow to start, memory hungry, and plagued by GC issues, JVM docker images are always huge. I don't want two layers of virtualization. If Docker and k8s become more mainstream that's probably bad news for a lot of JVM projects. Of course, Pulsar itself is quite old (2013?) so it was built well before any of the containerization stuff took off.
  - ckdarby 6y ago
    I don't understand the relationship between containers, kubernetes and JVM. I run a prod Pulsar cluster using helm charts. All containers & kubernetes, zero issue.
  - zumachase 6y ago
    I’m not really sure what containers/k8s has to do with JVM? Containers are just name spacing, not virtualization. If you’re running in the cloud, either way you slice it you’ll have two virtual machines: hypervisor and JVM. I’d argue that being able to herd your JVM procs like cattle makes them good candidates for k8s because you can always just set resource limits so they get purged when the heap becomes too large.
- q3k 6y ago
  > ... systems that rely on external dependencies like Apache Zookeeper and BookKeeper. I would argue that building up on existing systems (especially things like a distributed consensus system like Zookeeper) is generally good practice. Being able to just point a bunch of self-contained JARs/WARs at a Zookeeper/etcd cluster and not worry about startup sequencing or cluster bootstrapping makes me happy.
  - imglorp 6y ago
    Hasn't Kafka been talking about eliminating external zk forever? Are they making it an internal library or adopting a new consensus scheme?
    - miguno 6y ago
      Yes, you remember correctly. The Kafka community has already started the work on getting rid of ZooKeeper. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A... for details.
    - FridgeSeal 6y ago
      From my understanding, they’re going to use the brokers as their own metadata store, so the data that was in ZK goes in a topic instead, and I believe Raft for leadership? KIP-500 is the proposal/PR (?) for ZK-free Kafka which I understand has been merged in and work is now progressing on follow up tasks.
- vips7L 6y ago
  Modern JVMs don't take that long to start. While not at raw binary levels they shouldn't be killing your focus as theyre on par with scripting languages. https://cl4es.github.io/2019/11/20/OpenJDK-Startup-Update.html https://cl4es.github.io/2019/11/20/OpenJDK-Startup-Update.ht....
  - KMag 6y ago
    That depends on the filesystem, memory pressure, etc. Most JVM-based apps have quite a bit of filesystem I/O at startup, and a surprising memory footprint. If the kernel's page cache is mostly full of dirty pages, that's a bunch of extra disk writes in order to free up RAM for the JVM. These startup benchmarks are typically not run with the JVM and application jars being pulled from NFS or AFS on a machine with a lot of dirty pages in the page cache. Some small sh/bash app using curl or wget to hit a REST API is going to have a lot less network and disk I/O in that case, even if bash and curl are being pulled from NFS/AFS. Many companies run JVMs from NFS/AFS to simplify deployment and management. Also, if the OP is logged into a server running the JVM-based CLI app, they may be running it on a server JVM and getting much more eager JITting. Maybe it's not fair to the JVM that there are lots of circumstances where things are accidentally tuned poorly for the JVM. On the other hand, there is a lot to be said for tiny apps that are pretty resilient to poor conditions. Sometimes size still matters, even on big servers with plenty of RAM and cores.
miguno 6y ago
> The broker being stateless is great. I've had issues with moving partitions around in Kafka. Well, the Pulsar broker is (kinda) stateless, because they are essentially a caching layer in front of BookKeeper. But where's your data actually stored then? In BookKeeper bookies, which are stateful. Killing and replacing/restarting a Bookkeeper node requires the same redistribution of data as required in Kafka’s case. (Additionally, BookKeeper needs a separate data recovery daemon to be run and operated, https://bookkeeper.apache.org/archives/docs/r4.4.0/bookieRecovery.html https://bookkeeper.apache.org/archives/docs/r4.4.0/bookieRec...) So the comparison of 'Pulsar broker' vs. 'Kafka broker' is very misleading because, despite identical names, the respective brokers provide very different functionality. It's an apples-to-oranges comparison, like if you'd compare memcached (Pulsar broker) vs. Postgres (Kafka broker).
- ckdarby 6y ago
  Brokers in Pulsar can be seamlessly added or removed. Bookies can seamlessly be added. Kafka unless there has been a KIP since I stopped paying attention to Kafka doesn't do this. I remember adding brokers to Kafka and taking advantage of them on existing topics meant repartitioning which if I recall correctly breaks the golden ordering contract that most devs bank on. The data written to the partition will always be in the order for that partition itself.
  - enitihas 6y ago
    How do bookkeeper nodes handle addition then. AFAIK, if you add new storage nodes, whether bookkeeper or Kafka, there has to be some repartitioning, or else how will the new node be useful at all?
    - sciurus 6y ago
      https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works https://jack-vanlightly.com/blog/2018/10/2/understanding-how... goes into this. > The data of a given topic is spread across multiple Bookies. The topic has been split into Ledgers and the Ledgers into Fragments and with striping, into calculatable subsets of fragment ensembles. When you need to grow your cluster, just add more Bookies and they’ll start getting written to when new fragments are created. No more Kafka-style rebalancing required. However, reads and writes now have to jump around a bit between Bookies.
kh_hk 6y ago
I found redis pub/sub to fit most of the needs for a lightweight message passing system, if you do not need persistence.
lulf 6y ago
I’ll throw in another alternative to look at for those using Kubernetes: https://enmasse.io https://enmasse.io
maxdo 6y ago
You Probably just heard something about kubernetes and never really used it. Pulsar has its own helm chart. Installation in this cause is as trivial as any other product. It might require 1-2 gb of memory more but is it really critical ? Nats itself is never stable product , which is been completely rewritten several times. If you want same level or functionality is not on pair or you have to use another rewrite of it in deep alpha. Just try to do a basis negative acknowledgment on NATS :) it’s not there What I like about Nats it’s their dev focused approach. Clients are shining. Pulsar instead doesn’t have a decent support of nodejs
molszanski 6y ago
+1 for nats. It is amazing how quickly it went from "what is nats" to "that feature that required a message broker is complete". Good job nats team.
httgp 6y ago
There’s also Liftbridge (https://liftbridge.io/ https://liftbridge.io/) which is built to offer Kafka-like features on top of NATS. It just hit v1.0 and looks super promising. Full disclaimer: I worked on building a client library for it while it was in its early stages, and from my limited time with it, it was super easy to operate, very light on resource usage and stupid-fast.
- didip 6y ago
  Is liftbridge open source?
heipei 6y ago
Depending on your application architecture I'd look into nsq (https://nsq.io/ https://nsq.io/) which works quite differently from most message brokers in that you're supposed to run one instance of the broker per server so you always have one local instance that you can publish to, no matter the current network conditions.
RedShift1 6y ago
No love for MQTT as message bus in this space?
site-packages1 6y ago
Seconding this comment, I've had nothing but complexity added for not much benefit in this space by needing to _also_ manage Zookeeper. Would love something small/stateless that isn't necessarily "Yahoo Production Grade" and can service hundreds or thousands of requests per second. Will check out NATS.
dehrmann 6y ago
It sounds like you're gripe isn't actually with Java or even the memory footprint, is that's you don't want to manage Zookeeper in addition to managing Kafka.
- gas9S9zw3P9c 6y ago
  Yeah, more or less. I'm not complaining about the JVM directly, but rather about what typical Java software, of which Kafka is one example, looks like, at least in my mind. Whenever I see JVM I think: Horrible user experience and tooling, bad memory footprint, slow to start and deploy, complex codebase where it's impossible to debug issues, huge Docker images, complicated configuration and tuning options that doesn't work out of the box, dependency hell, and so on. It's the typical enterprise package for managers that don't want to get fired - Nobody got ever fired for buying IBM. When I see something like NATS - A few thousand lines of code and a 40MB image, that makes me happy. It's a lightweight thing that "just works." That's why I stay away from anything JVM. And I acknowledge that I am totally biased and may be wrong in some cases, but that has been my experience.
iampims 6y ago
For a drop-in Kafka replacement, check out: https://vectorized.io/redpanda/ https://vectorized.io/redpanda/

KptMarchewa 6y ago

>That being said, I'm getting tired of managing these clunky memory-hungry JVM-based systems that rely on external dependencies like Apache Zookeeper and BookKeeper. One of my pet peeves with Kafka were CLI tools written in JVM languages. Slow JVM startup time was killing my focus.

gas9S9zw3P9c 6y ago
Yep. I'm using containers and Kubernetes in production so I'm trying to stay far away from anything JVM-based. On top of being slow to start, memory hungry, and plagued by GC issues, JVM docker images are always huge. I don't want two layers of virtualization. If Docker and k8s become more mainstream that's probably bad news for a lot of JVM projects. Of course, Pulsar itself is quite old (2013?) so it was built well before any of the containerization stuff took off.
- ckdarby 6y ago
  I don't understand the relationship between containers, kubernetes and JVM. I run a prod Pulsar cluster using helm charts. All containers & kubernetes, zero issue.
- zumachase 6y ago
  I’m not really sure what containers/k8s has to do with JVM? Containers are just name spacing, not virtualization. If you’re running in the cloud, either way you slice it you’ll have two virtual machines: hypervisor and JVM. I’d argue that being able to herd your JVM procs like cattle makes them good candidates for k8s because you can always just set resource limits so they get purged when the heap becomes too large.
q3k 6y ago
> ... systems that rely on external dependencies like Apache Zookeeper and BookKeeper. I would argue that building up on existing systems (especially things like a distributed consensus system like Zookeeper) is generally good practice. Being able to just point a bunch of self-contained JARs/WARs at a Zookeeper/etcd cluster and not worry about startup sequencing or cluster bootstrapping makes me happy.
- imglorp 6y ago
  Hasn't Kafka been talking about eliminating external zk forever? Are they making it an internal library or adopting a new consensus scheme?
  - miguno 6y ago
    Yes, you remember correctly. The Kafka community has already started the work on getting rid of ZooKeeper. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A... for details.
  - FridgeSeal 6y ago
    From my understanding, they’re going to use the brokers as their own metadata store, so the data that was in ZK goes in a topic instead, and I believe Raft for leadership? KIP-500 is the proposal/PR (?) for ZK-free Kafka which I understand has been merged in and work is now progressing on follow up tasks.
vips7L 6y ago
Modern JVMs don't take that long to start. While not at raw binary levels they shouldn't be killing your focus as theyre on par with scripting languages. https://cl4es.github.io/2019/11/20/OpenJDK-Startup-Update.html https://cl4es.github.io/2019/11/20/OpenJDK-Startup-Update.ht....
- KMag 6y ago
  That depends on the filesystem, memory pressure, etc. Most JVM-based apps have quite a bit of filesystem I/O at startup, and a surprising memory footprint. If the kernel's page cache is mostly full of dirty pages, that's a bunch of extra disk writes in order to free up RAM for the JVM. These startup benchmarks are typically not run with the JVM and application jars being pulled from NFS or AFS on a machine with a lot of dirty pages in the page cache. Some small sh/bash app using curl or wget to hit a REST API is going to have a lot less network and disk I/O in that case, even if bash and curl are being pulled from NFS/AFS. Many companies run JVMs from NFS/AFS to simplify deployment and management. Also, if the OP is logged into a server running the JVM-based CLI app, they may be running it on a server JVM and getting much more eager JITting. Maybe it's not fair to the JVM that there are lots of circumstances where things are accidentally tuned poorly for the JVM. On the other hand, there is a lot to be said for tiny apps that are pretty resilient to poor conditions. Sometimes size still matters, even on big servers with plenty of RAM and cores.

miguno 6y ago

> The broker being stateless is great. I've had issues with moving partitions around in Kafka. Well, the Pulsar broker is (kinda) stateless, because they are essentially a caching layer in front of BookKeeper. But where's your data actually stored then? In BookKeeper bookies, which are stateful. Killing and replacing/restarting a Bookkeeper node requires the same redistribution of data as required in Kafka’s case. (Additionally, BookKeeper needs a separate data recovery daemon to be run and operated, https://bookkeeper.apache.org/archives/docs/r4.4.0/bookieRecovery.html https://bookkeeper.apache.org/archives/docs/r4.4.0/bookieRec...) So the comparison of 'Pulsar broker' vs. 'Kafka broker' is very misleading because, despite identical names, the respective brokers provide very different functionality. It's an apples-to-oranges comparison, like if you'd compare memcached (Pulsar broker) vs. Postgres (Kafka broker).

ckdarby 6y ago
Brokers in Pulsar can be seamlessly added or removed. Bookies can seamlessly be added. Kafka unless there has been a KIP since I stopped paying attention to Kafka doesn't do this. I remember adding brokers to Kafka and taking advantage of them on existing topics meant repartitioning which if I recall correctly breaks the golden ordering contract that most devs bank on. The data written to the partition will always be in the order for that partition itself.
- enitihas 6y ago
  How do bookkeeper nodes handle addition then. AFAIK, if you add new storage nodes, whether bookkeeper or Kafka, there has to be some repartitioning, or else how will the new node be useful at all?
  - sciurus 6y ago
    https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works https://jack-vanlightly.com/blog/2018/10/2/understanding-how... goes into this. > The data of a given topic is spread across multiple Bookies. The topic has been split into Ledgers and the Ledgers into Fragments and with striping, into calculatable subsets of fragment ensembles. When you need to grow your cluster, just add more Bookies and they’ll start getting written to when new fragments are created. No more Kafka-style rebalancing required. However, reads and writes now have to jump around a bit between Bookies.

kh_hk 6y ago

I found redis pub/sub to fit most of the needs for a lightweight message passing system, if you do not need persistence.

lulf 6y ago

I’ll throw in another alternative to look at for those using Kubernetes: https://enmasse.io https://enmasse.io

maxdo 6y ago

You Probably just heard something about kubernetes and never really used it. Pulsar has its own helm chart. Installation in this cause is as trivial as any other product. It might require 1-2 gb of memory more but is it really critical ? Nats itself is never stable product , which is been completely rewritten several times. If you want same level or functionality is not on pair or you have to use another rewrite of it in deep alpha. Just try to do a basis negative acknowledgment on NATS :) it’s not there What I like about Nats it’s their dev focused approach. Clients are shining. Pulsar instead doesn’t have a decent support of nodejs

molszanski 6y ago

+1 for nats. It is amazing how quickly it went from "what is nats" to "that feature that required a message broker is complete". Good job nats team.

httgp 6y ago

There’s also Liftbridge (https://liftbridge.io/ https://liftbridge.io/) which is built to offer Kafka-like features on top of NATS. It just hit v1.0 and looks super promising. Full disclaimer: I worked on building a client library for it while it was in its early stages, and from my limited time with it, it was super easy to operate, very light on resource usage and stupid-fast.

didip 6y ago
Is liftbridge open source?

heipei 6y ago

Depending on your application architecture I'd look into nsq (https://nsq.io/ https://nsq.io/) which works quite differently from most message brokers in that you're supposed to run one instance of the broker per server so you always have one local instance that you can publish to, no matter the current network conditions.

RedShift1 6y ago

No love for MQTT as message bus in this space?

site-packages1 6y ago

Seconding this comment, I've had nothing but complexity added for not much benefit in this space by needing to _also_ manage Zookeeper. Would love something small/stateless that isn't necessarily "Yahoo Production Grade" and can service hundreds or thousands of requests per second. Will check out NATS.

dehrmann 6y ago

It sounds like you're gripe isn't actually with Java or even the memory footprint, is that's you don't want to manage Zookeeper in addition to managing Kafka.

gas9S9zw3P9c 6y ago
Yeah, more or less. I'm not complaining about the JVM directly, but rather about what typical Java software, of which Kafka is one example, looks like, at least in my mind. Whenever I see JVM I think: Horrible user experience and tooling, bad memory footprint, slow to start and deploy, complex codebase where it's impossible to debug issues, huge Docker images, complicated configuration and tuning options that doesn't work out of the box, dependency hell, and so on. It's the typical enterprise package for managers that don't want to get fired - Nobody got ever fired for buying IBM. When I see something like NATS - A few thousand lines of code and a 40MB image, that makes me happy. It's a lightweight thing that "just works." That's why I stay away from anything JVM. And I acknowledge that I am totally biased and may be wrong in some cases, but that has been my experience.

iampims 6y ago

For a drop-in Kafka replacement, check out: https://vectorized.io/redpanda/ https://vectorized.io/redpanda/