20 ms·

>That being said, I'm getting tired of managing these clunky memory-hungry JVM-based systems that rely on external dependencies like Apache Zookeeper and BookKe

>That being said, I'm getting tired of managing these clunky memory-hungry JVM-based systems that rely on external dependencies like Apache Zookeeper and BookKeeper. One of my pet peeves with Kafka were CLI tools written in JVM languages. Slow JVM startup time was killing my focus.

gas9S9zw3P9c 6y ago
Yep. I'm using containers and Kubernetes in production so I'm trying to stay far away from anything JVM-based. On top of being slow to start, memory hungry, and plagued by GC issues, JVM docker images are always huge. I don't want two layers of virtualization. If Docker and k8s become more mainstream that's probably bad news for a lot of JVM projects. Of course, Pulsar itself is quite old (2013?) so it was built well before any of the containerization stuff took off.
- ckdarby 6y ago
  I don't understand the relationship between containers, kubernetes and JVM. I run a prod Pulsar cluster using helm charts. All containers & kubernetes, zero issue.
- zumachase 6y ago
  I’m not really sure what containers/k8s has to do with JVM? Containers are just name spacing, not virtualization. If you’re running in the cloud, either way you slice it you’ll have two virtual machines: hypervisor and JVM. I’d argue that being able to herd your JVM procs like cattle makes them good candidates for k8s because you can always just set resource limits so they get purged when the heap becomes too large.
q3k 6y ago
> ... systems that rely on external dependencies like Apache Zookeeper and BookKeeper. I would argue that building up on existing systems (especially things like a distributed consensus system like Zookeeper) is generally good practice. Being able to just point a bunch of self-contained JARs/WARs at a Zookeeper/etcd cluster and not worry about startup sequencing or cluster bootstrapping makes me happy.
- imglorp 6y ago
  Hasn't Kafka been talking about eliminating external zk forever? Are they making it an internal library or adopting a new consensus scheme?
  - miguno 6y ago
    Yes, you remember correctly. The Kafka community has already started the work on getting rid of ZooKeeper. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A... for details.
  - FridgeSeal 6y ago
    From my understanding, they’re going to use the brokers as their own metadata store, so the data that was in ZK goes in a topic instead, and I believe Raft for leadership? KIP-500 is the proposal/PR (?) for ZK-free Kafka which I understand has been merged in and work is now progressing on follow up tasks.
vips7L 6y ago
Modern JVMs don't take that long to start. While not at raw binary levels they shouldn't be killing your focus as theyre on par with scripting languages. https://cl4es.github.io/2019/11/20/OpenJDK-Startup-Update.html https://cl4es.github.io/2019/11/20/OpenJDK-Startup-Update.ht....
- KMag 6y ago
  That depends on the filesystem, memory pressure, etc. Most JVM-based apps have quite a bit of filesystem I/O at startup, and a surprising memory footprint. If the kernel's page cache is mostly full of dirty pages, that's a bunch of extra disk writes in order to free up RAM for the JVM. These startup benchmarks are typically not run with the JVM and application jars being pulled from NFS or AFS on a machine with a lot of dirty pages in the page cache. Some small sh/bash app using curl or wget to hit a REST API is going to have a lot less network and disk I/O in that case, even if bash and curl are being pulled from NFS/AFS. Many companies run JVMs from NFS/AFS to simplify deployment and management. Also, if the OP is logged into a server running the JVM-based CLI app, they may be running it on a server JVM and getting much more eager JITting. Maybe it's not fair to the JVM that there are lots of circumstances where things are accidentally tuned poorly for the JVM. On the other hand, there is a lot to be said for tiny apps that are pretty resilient to poor conditions. Sometimes size still matters, even on big servers with plenty of RAM and cores.

gas9S9zw3P9c 6y ago

Yep. I'm using containers and Kubernetes in production so I'm trying to stay far away from anything JVM-based. On top of being slow to start, memory hungry, and plagued by GC issues, JVM docker images are always huge. I don't want two layers of virtualization. If Docker and k8s become more mainstream that's probably bad news for a lot of JVM projects. Of course, Pulsar itself is quite old (2013?) so it was built well before any of the containerization stuff took off.

ckdarby 6y ago
I don't understand the relationship between containers, kubernetes and JVM. I run a prod Pulsar cluster using helm charts. All containers & kubernetes, zero issue.
zumachase 6y ago
I’m not really sure what containers/k8s has to do with JVM? Containers are just name spacing, not virtualization. If you’re running in the cloud, either way you slice it you’ll have two virtual machines: hypervisor and JVM. I’d argue that being able to herd your JVM procs like cattle makes them good candidates for k8s because you can always just set resource limits so they get purged when the heap becomes too large.

q3k 6y ago

> ... systems that rely on external dependencies like Apache Zookeeper and BookKeeper. I would argue that building up on existing systems (especially things like a distributed consensus system like Zookeeper) is generally good practice. Being able to just point a bunch of self-contained JARs/WARs at a Zookeeper/etcd cluster and not worry about startup sequencing or cluster bootstrapping makes me happy.

imglorp 6y ago
Hasn't Kafka been talking about eliminating external zk forever? Are they making it an internal library or adopting a new consensus scheme?
- miguno 6y ago
  Yes, you remember correctly. The Kafka community has already started the work on getting rid of ZooKeeper. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A... for details.
- FridgeSeal 6y ago
  From my understanding, they’re going to use the brokers as their own metadata store, so the data that was in ZK goes in a topic instead, and I believe Raft for leadership? KIP-500 is the proposal/PR (?) for ZK-free Kafka which I understand has been merged in and work is now progressing on follow up tasks.

vips7L 6y ago

Modern JVMs don't take that long to start. While not at raw binary levels they shouldn't be killing your focus as theyre on par with scripting languages. https://cl4es.github.io/2019/11/20/OpenJDK-Startup-Update.html https://cl4es.github.io/2019/11/20/OpenJDK-Startup-Update.ht....

KMag 6y ago
That depends on the filesystem, memory pressure, etc. Most JVM-based apps have quite a bit of filesystem I/O at startup, and a surprising memory footprint. If the kernel's page cache is mostly full of dirty pages, that's a bunch of extra disk writes in order to free up RAM for the JVM. These startup benchmarks are typically not run with the JVM and application jars being pulled from NFS or AFS on a machine with a lot of dirty pages in the page cache. Some small sh/bash app using curl or wget to hit a REST API is going to have a lot less network and disk I/O in that case, even if bash and curl are being pulled from NFS/AFS. Many companies run JVMs from NFS/AFS to simplify deployment and management. Also, if the OP is logged into a server running the JVM-based CLI app, they may be running it on a server JVM and getting much more eager JITting. Maybe it's not fair to the JVM that there are lots of circumstances where things are accidentally tuned poorly for the JVM. On the other hand, there is a lot to be said for tiny apps that are pretty resilient to poor conditions. Sometimes size still matters, even on big servers with plenty of RAM and cores.