6 ms·

Coincidentally, as someone who worked on this blog, I was surprised (and pleased!) to see that we are not the only ones who felt the need to build a Zookeeper a

Coincidentally, as someone who worked on this blog, I was surprised (and pleased!) to see that we are not the only ones who felt the need to build a Zookeeper alternative. Looks like folks at StreamNative did as well, with their Oxia project: https://github.com/streamnative/oxia https://github.com/streamnative/oxia. They were just talking about this yesterday at Confluent Current ("Introducing Oxia: A Scalable Zookeeper Alternative" was the title of their talk). https://streamnative.io/blog/introducing-oxia-scalable-metadata-and-coordination https://streamnative.io/blog/introducing-oxia-scalable-metad... Seems to be a trend :)

kapilvt 3y ago
looks to be a slightly different design goal on oxia wrt to replication and fault tolerance. https://github.com/streamnative/oxia/blob/main/docs/design-goals.md#replication-protocol https://github.com/streamnative/oxia/blob/main/docs/design-g...
- tylerhannan 3y ago
  Def. I hadn't seen Oxia before but the idea, for their implementation, of making Zookeeper more like Bookkeeper was an interesting one. Not right for ClickHouse needs but, IMO, a novel approach.
coding123 3y ago
Is the trend mainly due to ZK being written in Java?
- tylerhannan 3y ago
  TBH, I don't think so. I mean, I have worked and, and been guilty of tooling driven development (RiiR anyone?) . But, also, in a comment below Alexey shares many of the reasons other than language. I think Oxia does a good job of sharing their approach in - https://github.com/streamnative/oxia/blob/main/docs/design-goals.md https://github.com/streamnative/oxia/blob/main/docs/design-g... (Alexey's comment, FYI, https://news.ycombinator.com/item?id=37677324 https://news.ycombinator.com/item?id=37677324)
- Xeoncross 3y ago
  It's not the written in Java part, it's the running in the JVM that's the issue. Memory hungry is what I think of when I think Java apps. I'd much rather have a low memory Go or Rust service.
  - adra 3y ago
    I mean java native compilation is a thing. I imagine you'd get the majority of the lift via that route instead of reinventing the wheel, but it's possible there are caveats to that route for these impls. Even java native/golang have GC heaps, but in general ZK is really low GC churn so I don't see that being the impetus to rewrite. My guess if anything is that people always complain about ZK being annoying as a second piece of infra to distribute for simple setups, so my guess this is just a prelude to them embedding their keeper into the DB deployment itself which is the same general strategy Kafka is taking (a verrrry loong time) to rollout.
  - dathinab 3y ago
    but if you use ZK for what it was designed for (basic cluster role/naming cordination, distributing configs to cluster nodes and similar) then this kind doesn't matter I mean in a typical use case of it you would - run it on long running nodes (e.g. not lambda or spot instances) - run more or less exactly 3 nodes up to quite a cluster size, I guess some use-cases which involve a lot of serverless might need more - configs tend to not change "that" much nor are they "that" big what this means is - java needing time to run-hot (JIT optimize) is not an issue for it - GC isn't an issue and if you look at how much memory (RAM) typical minimal nodes in the cloud have it in context of typical config sizes and cluster sizes is also not an issue through I guess depending what you want to do there could be issues if you use it - for squeezing through analytics or similar - setups with very very very large constantly changing clusters, e.g. in some serverless context with ton of ad-hoc spawned up instances, maybe using WASM and certain snapshot tricks allowing insane fast startup time - you want to bundle it directly into other applications, running on the same hardware and that applications need more memory but all of it are cases it wasn't designed for so I wouldn't call it an "alternative" but a ZooKeeper like service for different use-cases, I guess
Lt_Riza_Hawkeye 3y ago
Meta additionally has some internal software called Zelos, which is the zookeeper api but implemented on delos[0], though you didn't hear it from me [0] https://www.usenix.org/system/files/osdi20-balakrishnan.pdf https://www.usenix.org/system/files/osdi20-balakrishnan.pdf

kapilvt 3y ago

looks to be a slightly different design goal on oxia wrt to replication and fault tolerance. https://github.com/streamnative/oxia/blob/main/docs/design-goals.md#replication-protocol https://github.com/streamnative/oxia/blob/main/docs/design-g...

tylerhannan 3y ago
Def. I hadn't seen Oxia before but the idea, for their implementation, of making Zookeeper more like Bookkeeper was an interesting one. Not right for ClickHouse needs but, IMO, a novel approach.

coding123 3y ago

Is the trend mainly due to ZK being written in Java?

tylerhannan 3y ago
TBH, I don't think so. I mean, I have worked and, and been guilty of tooling driven development (RiiR anyone?) . But, also, in a comment below Alexey shares many of the reasons other than language. I think Oxia does a good job of sharing their approach in - https://github.com/streamnative/oxia/blob/main/docs/design-goals.md https://github.com/streamnative/oxia/blob/main/docs/design-g... (Alexey's comment, FYI, https://news.ycombinator.com/item?id=37677324 https://news.ycombinator.com/item?id=37677324)
Xeoncross 3y ago
It's not the written in Java part, it's the running in the JVM that's the issue. Memory hungry is what I think of when I think Java apps. I'd much rather have a low memory Go or Rust service.
- adra 3y ago
  I mean java native compilation is a thing. I imagine you'd get the majority of the lift via that route instead of reinventing the wheel, but it's possible there are caveats to that route for these impls. Even java native/golang have GC heaps, but in general ZK is really low GC churn so I don't see that being the impetus to rewrite. My guess if anything is that people always complain about ZK being annoying as a second piece of infra to distribute for simple setups, so my guess this is just a prelude to them embedding their keeper into the DB deployment itself which is the same general strategy Kafka is taking (a verrrry loong time) to rollout.
- dathinab 3y ago
  but if you use ZK for what it was designed for (basic cluster role/naming cordination, distributing configs to cluster nodes and similar) then this kind doesn't matter I mean in a typical use case of it you would - run it on long running nodes (e.g. not lambda or spot instances) - run more or less exactly 3 nodes up to quite a cluster size, I guess some use-cases which involve a lot of serverless might need more - configs tend to not change "that" much nor are they "that" big what this means is - java needing time to run-hot (JIT optimize) is not an issue for it - GC isn't an issue and if you look at how much memory (RAM) typical minimal nodes in the cloud have it in context of typical config sizes and cluster sizes is also not an issue through I guess depending what you want to do there could be issues if you use it - for squeezing through analytics or similar - setups with very very very large constantly changing clusters, e.g. in some serverless context with ton of ad-hoc spawned up instances, maybe using WASM and certain snapshot tricks allowing insane fast startup time - you want to bundle it directly into other applications, running on the same hardware and that applications need more memory but all of it are cases it wasn't designed for so I wouldn't call it an "alternative" but a ZooKeeper like service for different use-cases, I guess

Lt_Riza_Hawkeye 3y ago

Meta additionally has some internal software called Zelos, which is the zookeeper api but implemented on delos[0], though you didn't hear it from me [0] https://www.usenix.org/system/files/osdi20-balakrishnan.pdf https://www.usenix.org/system/files/osdi20-balakrishnan.pdf