12 ms·

What does zookeeper even do? I know it's supposed to be a key value store. But the only thing i ever hear about zookeeper is not people actually doing stuff wit

What does zookeeper even do? I know it's supposed to be a key value store. But the only thing i ever hear about zookeeper is not people actually doing stuff with it, but rather doing stuff for it. It seems like the most useless piece of software. Like someone wrote a concept down but said it was software.

brodouevencode 3y ago
There are good use cases for it. If you have a cluster (or clusters) with many, many nodes running your application you need to be able to communicate changes/configurations to those nodes/applications. Zookeeper is really good at doing this because it itself is clustered (HA, redundant). Updating all of those nodes/applications manually is not a desirable thing, so you have the application call the zookeeper service to get those configuration changes. The problem I see often is that it's wedged into places for which it's absolutely overkill. As in clusters with less than a dozen nodes, no service level determining such a need, etc. - it's often just a shiny object for architects.
- mdaniel 3y ago
  > you have the application call the zookeeper service to get those configuration changes. as a note, both ZK and etcd have key watches such that one need not poll the key-value store, it'll push out notifications for changes to subscribers, making for super quick reactions in any such distributed applications
elric 3y ago
It basically offers a set of distributed primitives which lets you build distributed applications with far fewer headaches. To be fair, it's mostly pretty good at what it does, but it's not super ops-friendly, and when things do go wrong, recovery isn't a whole lot of fun. Apache Curator builds on top of ZooKeeper and turns the primitives into usable recipes, which is what I recommend to most people these days.
- tetha 3y ago
  Like, I'm a consul user, but I'd assume the use cases are similar. One concrete example: Consul can provide you with locking and thus leader election with relatively little effort. Patroni, the postgres manager, can use a consul lock on a key to elect the postgres leader and consul ensures mutual exclusion. Or in a similar way, I needed to run some data collection once a day, but the cluster consisted of 3 identical nodes and I didn't want to start treating one node specially. I could use consul to hold a lock by using <consul lock whatever/lock collect-data.sh>. Once I had mutual exclusion, I could store the last time data was collected in a consul kv entry with <consul kv put whatever/last-run $( date +"%s")>. Then each script just grabbed the last time data was collected or 0 if missing and checked if it was 23ish hours ago. And there we go, all systems identical and fault tolerant once-a-day data collection without additional config to forget on top.
chatmasta 3y ago
I've thus far avoided using it, but my recollection is that it kinda snuck into mainstream stacks because it was part of the standard deployment model for the ~2015 era of "big data," with Hadoop, HBase, Solr, and the other usual suspects from the Apache Java ecosystem. Nobody chose to have Zookeeper in their stack, but they needed it for these other services and so eventually they were stuck with it.

brodouevencode 3y ago

There are good use cases for it. If you have a cluster (or clusters) with many, many nodes running your application you need to be able to communicate changes/configurations to those nodes/applications. Zookeeper is really good at doing this because it itself is clustered (HA, redundant). Updating all of those nodes/applications manually is not a desirable thing, so you have the application call the zookeeper service to get those configuration changes. The problem I see often is that it's wedged into places for which it's absolutely overkill. As in clusters with less than a dozen nodes, no service level determining such a need, etc. - it's often just a shiny object for architects.

mdaniel 3y ago
> you have the application call the zookeeper service to get those configuration changes. as a note, both ZK and etcd have key watches such that one need not poll the key-value store, it'll push out notifications for changes to subscribers, making for super quick reactions in any such distributed applications

elric 3y ago

It basically offers a set of distributed primitives which lets you build distributed applications with far fewer headaches. To be fair, it's mostly pretty good at what it does, but it's not super ops-friendly, and when things do go wrong, recovery isn't a whole lot of fun. Apache Curator builds on top of ZooKeeper and turns the primitives into usable recipes, which is what I recommend to most people these days.

tetha 3y ago
Like, I'm a consul user, but I'd assume the use cases are similar. One concrete example: Consul can provide you with locking and thus leader election with relatively little effort. Patroni, the postgres manager, can use a consul lock on a key to elect the postgres leader and consul ensures mutual exclusion. Or in a similar way, I needed to run some data collection once a day, but the cluster consisted of 3 identical nodes and I didn't want to start treating one node specially. I could use consul to hold a lock by using <consul lock whatever/lock collect-data.sh>. Once I had mutual exclusion, I could store the last time data was collected in a consul kv entry with <consul kv put whatever/last-run $( date +"%s")>. Then each script just grabbed the last time data was collected or 0 if missing and checked if it was 23ish hours ago. And there we go, all systems identical and fault tolerant once-a-day data collection without additional config to forget on top.

chatmasta 3y ago

I've thus far avoided using it, but my recollection is that it kinda snuck into mainstream stacks because it was part of the standard deployment model for the ~2015 era of "big data," with Hadoop, HBase, Solr, and the other usual suspects from the Apache Java ecosystem. Nobody chose to have Zookeeper in their stack, but they needed it for these other services and so eventually they were stuck with it.