7 ms·

ZK is vastly more popular, it's just much less sexy so people don't write about it. Every production Kafka, Spark, Hadoop, Pulsar, etc cluster is using ZK. Just

by jpgvm 5y ago

ZK is vastly more popular, it's just much less sexy so people don't write about it. Every production Kafka, Spark, Hadoop, Pulsar, etc cluster is using ZK. Just the Kafka and Hadoop clusters alone probably allows it to rival etcd in footprint. Consul on the other hand isn't used widely outside of some startups and a few OSS systems like Grafana Cortex/Loki. When it comes to which one has the most production hours under demanding workloads I think ZK comes out miles ahead of everything, then etcd (because of k8s) then in a very distant third is Consul.

Thaxll 5y ago
Lot of people moved away from ZK because it used to be a nightmare to setup and maintain, ZK is not vastly more popular than Consul. In every places I worked in the last 5y we did not even considered ZK because of the setup and all the components it requires.
sofixa 5y ago
ZK isn't sexy, we agree. IIRC Kafka no longer requires it? And IIRC it was one of the main downsides of Kafka, that it required a whole ZK cluster. etcd doesn't compare directly to ZK and Consul, because it's just a distributed KV store. You can build service discovery on top of it with custom tooling, but doesn't come out of the box. Consul does that + service discovery + service mesh + (since last week ) API gateway. On one hand, it's lots of stuff packed together, on the other hand it's useful. I started with it for SD, but when KV was also needed it was easy to add. I'm not too familiar with ZK, but isn't it also mostly a distributed KV store? Does it have things like Consul's health checks, prepared queries, DNS interface, etc? Consul is more widely used than that, like Cloudflare and Criteo at huge scale.
- jpgvm 5y ago
  Kafka still requires it, at least for all production clusters. In a few years the recommendation might be the new internal Raft based consensus though.
0xbadcafebee 5y ago
Consul is at least as widely used as ZK now seeing as anyone who wants to run a distributed system on Hashi tools needs to use it. It's much better designed, and easier to operate, than ZK. ZK, Solr, and other apache projects are pretty terrible in comparison.

by jpgvm 5y ago

Thaxll 5y ago

Lot of people moved away from ZK because it used to be a nightmare to setup and maintain, ZK is not vastly more popular than Consul. In every places I worked in the last 5y we did not even considered ZK because of the setup and all the components it requires.

sofixa 5y ago

ZK isn't sexy, we agree. IIRC Kafka no longer requires it? And IIRC it was one of the main downsides of Kafka, that it required a whole ZK cluster. etcd doesn't compare directly to ZK and Consul, because it's just a distributed KV store. You can build service discovery on top of it with custom tooling, but doesn't come out of the box. Consul does that + service discovery + service mesh + (since last week ) API gateway. On one hand, it's lots of stuff packed together, on the other hand it's useful. I started with it for SD, but when KV was also needed it was easy to add. I'm not too familiar with ZK, but isn't it also mostly a distributed KV store? Does it have things like Consul's health checks, prepared queries, DNS interface, etc? Consul is more widely used than that, like Cloudflare and Criteo at huge scale.

jpgvm 5y ago
Kafka still requires it, at least for all production clusters. In a few years the recommendation might be the new internal Raft based consensus though.

0xbadcafebee 5y ago

Consul is at least as widely used as ZK now seeing as anyone who wants to run a distributed system on Hashi tools needs to use it. It's much better designed, and easier to operate, than ZK. ZK, Solr, and other apache projects are pretty terrible in comparison.