8 ms·
ClickHouse Keeper: A ZooKeeper alternative written in C++
- tylerhannan 3y agoThanks for sharing! If anyone has any questions, I'll do my best to get them answered. (Disclaimer: I work at ClickHouse)
- deleted 3y ago[deleted]
- alchemist1e9 3y agoIs there a python client library you can recommend?
- alesapin 3y agoAll ZooKeeper libraries are compatible with clickhouse-keeper. The most popular and mature is https://kazoo.readthedocs.io/en/latest/ https://kazoo.readthedocs.io/en/latest/. We use it in our integration tests framework (with clickhouse-keeper) a lot.
- zX41ZdbW 3y agoThe same library that you use for ZooKeeper - kazoo. Note: our stress tests have found a segmentation fault in Python's kazoo library. We only wanted to test Keeper, but found every bug around it :) Let me find a link.
- secondcoming 3y agoWhat do you use for network stuff in C++, ASIO?
- alesapin 3y agoYes, for internal RAFT implementation boost.asio is used.
- pdeva1 3y ago1. can this be used without clickhouse as just a zookeeper replacement? 2. am i correct in that its using s3 as disk? so can it be run as stateless pods in k8s? 3. if it uses s3, how are latency and costs of PUTs affected? does every write result in a PUT call to s3?
- alesapin 3y ago1. Absolutely. clickhouse-keeper is distributed as a standalone static binary or .deb package or .rpm package. You can use it without clickhouse as ZooKeeper replacement. 2. It's not recommended to use slow storage devices for logs in any coordination system (zookeeper, clickhouse-keeper, etcd and so on). Good setup will be small fast SSD/EBS disk for fresh logs and old logs + snapshots offloaded to S3. In such setup the amount of PUT requests will be tiny and latency will be as good as possible.
- pradeepchhetri 3y agoSometime back, I tried using clickhouse-keeper as zookeeper alternative with few other systems like kafka, mesos, solr, Wrote some notes here: https://pradeepchhetri.xyz/clickhousekeeper/ https://pradeepchhetri.xyz/clickhousekeeper/
- zX41ZdbW 3y ago1. Yes, it can be used with other applications as a ZooKeeper replacement, unless some unusual ZooKeper features are used (there is no Kerberos integration in Keeper, and it does not support the TTL of persistent nodes) or the application tests for a specific ZooKeeper version. 2. It could be configure to store - snapshots; - RAFT logs other than the latest log; in S3. It cannot use a stateless Kubernetes pod - the latest log has to be located on the filesystem. Although I see you can make a multi-region setup with multiple independent Kubernetes clusters and store logs in tmpfs (which is not 100% wrong from a theoretical standpoint), it is too risky to be practical. 3. Only the snapshots and the previous logs could be on S3, so the PUT requests are done only on log rotation.
- pdeva1 3y ago2. ok. so can i rebuild a cluster with just state in s3? eg: i create a cluster with local disks and s3 backing. entire cluster gets deleted. if i recreate cluster and point to same s3 bucket, will it restore its state?
- abronan 3y agoThanks for this excellent article! Enjoyed it from start to finish. This gave me a good memory of the work we've done at docker embedding our own replicated and consistent metadata storage using etcd's raft library. Looking at the initial pull request, is it correct that ClickHouse Keeper is based on Ebay's NuRaft library? Or did the Clickhouse team fork and modified this library to accommodate for ClickHouse usage and performance needs?
- alesapin 3y agoYes, you are right ClickHouse Keeper is based on NuRaft. We did a lot of modifications for this library, both for correctness and performance. Almost all of them (need to check) are contributed back to upstream ebay/NuRaft library.
- pram 3y agoLooks nice, I will definitely be trying this out. Built in s3 storage immediately sold me. I’ve used something called Exhibitor to manage ZK clusters in the past but it’s totally dead. Working with ZK is probably one of my least favorite things to do.
- randomtb 3y agoI used exhibitor in the past too. They were specially useful during the time when zookeeper cluster needed to expand/shrink or move host nodes. Zookeeper dynamic configuration solved that problem, which seems to be also supported by clickhouse keeper, Pretty impressive! Would definitely give it a try.
- tylerhannan 3y agoIt does indeed. Do note the docs page... https://clickhouse.com/docs/en/guides/sre/keeper/clickhouse-keeper#reconfiguration https://clickhouse.com/docs/en/guides/sre/keeper/clickhouse-... In particular, it is necessary to enable the `keeper_server.enable_reconfiguration` flag. It is pretty exhaustive coverage but if there is an important use case missing, let us know!
- lambda_garden 3y agoWhy is "written in C++" part of the headline? As engineers we focus too much on the implementation details and not the benefits to the user. How about: - ZooKeeper alternative with lower latency - ZooKeeper alternative with lower memory use - ZooKeeper alternative with predictable overheads (I don't know if these are true, just suggestions)
- GauntletWizard 3y agoI'd be most interested in a Zookeeper alternative that doesn't have massive bugs in leader election.
- abronan 3y agoSince the article states that they're using Raft and not ZAB for the consensus algorithm and leader election, it must be less prone to bugs when it comes to electing a leader. Since Raft is easier to reason about and the leader election process is more straightforward (Raft minimizes the chance that any two nodes will be candidates at the same time and thus avoids starting multiple concurrent elections).
- antonio2368 3y agoJust adding on to a nice response from abronan, our internal protocol also does some optimizations when it comes to leader election, e.g. Pre Vote protocol (https://github.com/eBay/NuRaft/blob/master/docs/prevote_protocol.md https://github.com/eBay/NuRaft/blob/master/docs/prevote_prot...) Also, we apply many different faults in our Jepsen tests which are run 3 times a day and we never had a problem with leader election. I know this doesn't confirm that there is no bug in it but it's pretty reassuring I would say.
- tbragin 3y agoDisclaimer: I worked on this blog with the team at ClickHouse. I like your suggestions! Some of the benefits we summarized in this summary page https://clickhouse.com/clickhouse/keeper https://clickhouse.com/clickhouse/keeper include ease of setup and operation, no overflow issues, better compression, faster recovery, (dramatically) less memory used, etc.. There was actually a reason why C++ was important for us at ClickHouse, and it's because C++ is our main code base and managing a Java project as part of it was not natural, but you right - for standalone use of this alternative, that doesn't matter.
- tbragin 3y agoCoincidentally, as someone who worked on this blog, I was surprised (and pleased!) to see that we are not the only ones who felt the need to build a Zookeeper alternative. Looks like folks at StreamNative did as well, with their Oxia project: https://github.com/streamnative/oxia https://github.com/streamnative/oxia. They were just talking about this yesterday at Confluent Current ("Introducing Oxia: A Scalable Zookeeper Alternative" was the title of their talk). https://streamnative.io/blog/introducing-oxia-scalable-metadata-and-coordination https://streamnative.io/blog/introducing-oxia-scalable-metad... Seems to be a trend :)
- kapilvt 3y agolooks to be a slightly different design goal on oxia wrt to replication and fault tolerance. https://github.com/streamnative/oxia/blob/main/docs/design-goals.md#replication-protocol https://github.com/streamnative/oxia/blob/main/docs/design-g...
- tylerhannan 3y agoDef. I hadn't seen Oxia before but the idea, for their implementation, of making Zookeeper more like Bookkeeper was an interesting one. Not right for ClickHouse needs but, IMO, a novel approach.
- coding123 3y agoIs the trend mainly due to ZK being written in Java?
- tylerhannan 3y agoTBH, I don't think so. I mean, I have worked and, and been guilty of tooling driven development (RiiR anyone?) . But, also, in a comment below Alexey shares many of the reasons other than language. I think Oxia does a good job of sharing their approach in - https://github.com/streamnative/oxia/blob/main/docs/design-goals.md https://github.com/streamnative/oxia/blob/main/docs/design-g... (Alexey's comment, FYI, https://news.ycombinator.com/item?id=37677324 https://news.ycombinator.com/item?id=37677324)
- jzelinskie 3y agoIt's been a few years since I've checked in with distributed lock services. Why would someone adopt ZooKeeper after etcd gained maturity? I recall seeing benchmarks more than 5 years ago where a naive proxy like zetcd[0] out-performs ZooKeeper itself in many ways and offers more consistent latencies. etcd has gotten lots of battle-testing being Kubernetes' datastore, but I can also see how that has shaped its design in a way that might not fit other projects. I think there are plenty of other projects (e.g. FoundationDB, Kafka) that also replaced their usage of ZooKeeper as their systems matured. I guess I'm confused why anyone has been picking up new installations of ZooKeeper. [0]: https://github.com/etcd-io/zetcd https://github.com/etcd-io/zetcd
- zX41ZdbW 3y agoThere is no specific reason to start with ZooKeeper, nor with ClickHouse Keeper, if you want to use another distributed consensus system. But: every such system is slightly different in the data model and the set of available primitives. It's very hard to build a distributed system correctly, even relying on ZooKeeper/Etcd/FoundationDB. For example, when I hear "distributed lock," I know that there is 90% chance there is a bug (distributed lock can be safely used if every transaction made under a lock also atomically tests that the lock still holds). So, if there is an existing system heavily relying on one distributed consensus implementation, it's very hard to switch to another. The main value of ClickHouse Keeper is its compatibility with ZooKeeper - it uses the same data model and wire protocol.
- endisneigh 3y agoFoundationdb doesn’t use locks, due to that it’s relatively easy to build distributed systems on top, but the trade off are 5sec transaction limits.
- klysm 3y agoThe term “distributed lock” is a bit of a mental red flag to me.
- Dowwie 3y agoAny thoughts here on Fly's Corrosion? https://github.com/superfly/corrosion https://github.com/superfly/corrosion
- mdaniel 3y agoAt least two comments spring to mind: this is at least _blogged_ as a drop-in ZK replacement, which for sure is not true of Corrosion, and ClickHouse has Jepsen tests for their distributed KV store, which I don't see any reference to such a thing for Corrosion Maybe neither of those two things matter for one's use case, but it's similar to someone rolling up on this blog post and saying "but what about etcd" -- they're just different, with wholly different operational and consumer concerns
- kiratp 3y agoWritten in C++ is not a positive in my book. New things created this decade in unsafe languages (where safe options would have worked fine) should be frowned up and criticized as bad engineering.
- krvajal 3y ago> where safe options would have worked fine Thats a very short-sighted view IMO, software engineering is not just about technology choices.
- kiratp 3y agoSoftware engineering is first and foremost about making good engineering choices. We are proving time and again as an industry that humans CANNOT write mistake-free code. From just this week: https://news.ycombinator.com/item?id=37600852 https://news.ycombinator.com/item?id=37600852 Just because someone built a fantastically functional building doesn't mean we can't criticize their choice of foundation. Case in point: Millennium Tower in SF: https://www.nbcbayarea.com/investigations/series/millennium-tower/san-francisco-millennium-tower-more-tilting/3249034/ https://www.nbcbayarea.com/investigations/series/millennium-...
- kgeist 3y agoClickhouse is a Yandex project and at Yandex, they historically use C++ for almost everything, I guess it's part of their culture (probably the founders were C++ programmers?) Their web services such as Yandex Taxi's backend (Uber's equivalent) are also written in C++ which is unusual for webdev nowadays.
- tylerhannan 3y agos/is a/was a/g But more interesting, to me, is language adoption and familiarity by region. I have a bookmarked dev.to article from 2020 that discussed programming language popularity by state - https://dev.to/eduecosystem/what-is-the-most-popular-programming-language-in-each-us-state-ha https://dev.to/eduecosystem/what-is-the-most-popular-program... I'm uncertain if anyone has extrapolated that to more geographic regions. It would be interesting.
- klysm 3y agoI hate running services in Java but this will have to earn a lot of trust before it’s a viable replacement in prod
- tylerhannan 3y ago++ agreed. ClickHouse Keeper was released as feature complete in December of 2021. It runs thousands of clusters, daily, both in CSP hosted offerings (including our own ClickHouse Cloud) and at customers running the OSS release. Never accept any claims at face value and always test. But, in this case, it is quite battle-hardened (i.e. the Jepsen tests run 3x daily https://github.com/ClickHouse/ClickHouse/tree/master/tests/jepsen.clickhouse https://github.com/ClickHouse/ClickHouse/tree/master/tests/j...).
- klysm 3y agoThat’s a strong endorsement. I wonder if there’s been any effect where it’s strongly tailored to the API surface area utilized by ClickHouse and there’s any gaps elsewhere
- tylerhannan 3y agoWe hope not and try to keep in wire compatible for clients to interact (recently added dynamic reconfig, etc.) It is definitely opinionated and influenced by our work...but not designed solely for it. But, also, we continue to improve. Most notably in the work on Multi-group Raft - https://github.com/ClickHouse/ClickHouse/issues/54172 https://github.com/ClickHouse/ClickHouse/issues/54172
- snotrockets 3y agoEspecially as it doesn't have memory safety, which is table stakes in 2023.
- spullara 3y agoYet another thing I have just used FoundationDB for in the past.
- rad_gruchalski 3y agoI’d love to read about that.
- davideberdin 3y agoI'm always impressed by the quality of the blog posts coming out from clickhouse.com! Super well written!
- tylerhannan 3y agoThank you! We try ;)
- macintux 3y agoBasho blog quality 4ever.
- tylerhannan 3y agoIt's a deep-cut reference. But it's one I am honoured you made. :mug:
- antonio2368 3y agoAs one of the contributors, I'm always happy to see interest and people using it. Keeper is a really interesting challenge and we're really open to any kind of feedback and thoughts. If you tried it out and have some feedback for it, I encourage you to create an issue (https://github.com/ClickHouse/ClickHouse https://github.com/ClickHouse/ClickHouse), ask on our Slack, ping me directly on Slack... (just don't call me on my phone) And don't forget that it's completely open-source like ClickHouse so contributors are more than welcome.
- Redsquare 3y agoshared mergetree is not open!
- xyzelement 3y agoI usually scoff at "written in.." part of such announcements, because it is a sign that the author is focused on the input ("I wrote this in X") not the output (value the user gets) In this case though, the blog outlines specific reasons why this had to be in C++ (interoperability w. their C++ codebase) as well as benefits that are separate from the language.
- The_Colonel 3y agoIt's a huge turn-off for me as well, because I interpret it as the main value it's supposed to deliver (which for me is 0 for the most part). Not talking about this specific project, just generally.
- hodgesrm 3y agoIn this case written in C++ is goodness. Also, having it be a variant of ClickHouse server that can run embedded or standalone is quite nice.
- lsofzz 3y ago> author is focused on the input ("I wrote this in X") not the output (value the user gets) Possibly, a lot of us enjoy developing X in Y for the sake of doing so. Not everyone may end up caring about the value that the user gets.
- dathinab 3y agoIs it just me or does it look like an "alternative for use-cases ZooKeeper was not intended for"? E.g. if we quote ZooKeeper: > ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. and ClickHouse > ClickHouse is the fastest and most resource-efficient open-source database for real-time applications and analytics. Like this are completely different use cases with just a small overlap.
- advisedwang 3y agoAnd then further down: > ClickHouse Keeper is a drop-in replacement for ZooKeeper That opening was about ClickHouse in general, but the article is about one particular application using the database.
- tylerhannan 3y agoYep. Generally, ClickHouse Keeper provides the coordination system for data replication and distributed DDL query execution for ClickHouse clusters.
- dathinab 3y agodrop in replacement doesn't change anything about my question, interfaces are not all what matters and now knowing more about it I would say the answer to my question is a clear yes, it used ZK for something ZK wasn't at all intended for which means it makes a lot of sense that they replace it
- macksd 3y agoAre you perhaps confusing ClickHouse, with ClickHouse Keeper (one of ClickHouse's components?) Sounds to me like ClickHouse is the database, and ClickHouse Keeper is the ZK drop-in replacement. A bit like HBase being a database, and ZooKeeper being a service it is heavily dependent on.
- dathinab 3y ago
- what-no-tests 3y agoI've been looking at RedPanda[0] for a new project. Any ideas on that? [0] https://redpanda.com/ https://redpanda.com/
- The_Colonel 3y agoDoes ClickHouse still have some relationship to Yandex?
- einpoklum 3y agoIt seems like these days, Yandex is just one of multiple stock holders; see: https://en.wikipedia.org/wiki/ClickHouse https://en.wikipedia.org/wiki/ClickHouse
- tylerhannan 3y ago"ClickHouse, Inc. is a Delaware company with headquarters in the San Francisco Bay Area. We have no operations in Russia, no Russian investors, and no Russian members of our Board of Directors." Source - https://clickhouse.com/blog/we-stand-with-ukraine https://clickhouse.com/blog/we-stand-with-ukraine
- einpoklum 3y ago"A Delaware company" ... funny :-) Also scary how disciplined US society is, with everyone informally-required to actively cheer the proxy war (while no company "stands with Yemen" for example).
- cvccvroomvroom 3y agoCompatible possibly but unproven in production at scale like ZK.
- tylerhannan 3y agoDefinitely used in production ;) and at rather some scale. It runs thousands of clusters, daily, both in CSP hosted offerings (including our own ClickHouse Cloud) and at customers running the OSS release. Never accept any claims at face value and always test. But, in this case, it is quite battle-hardened (i.e. the Jepsen tests run 3x daily https://github.com/ClickHouse/ClickHouse/tree/master/tests/jepsen.clickhouse https://github.com/ClickHouse/ClickHouse/tree/master/tests/j...) But yes, ZooKeeper is pretty amazing. We are building on the backs of giants. I'd also argue the RAFT v. ZAB is an important production scale conversation. But, as the blog says, Zookeper is a better option when you require scalability with a read-heavy workload.
- insanitybit 3y agoSo could I just point my Kafka at this thing and use it?
- tylerhannan 3y agofundamentally, yep. https://pradeepchhetri.xyz/clickhousekeeper/ https://pradeepchhetri.xyz/clickhousekeeper/ talks about some experiments in exactly that vein.
- qoega 3y agoYou can even migrate your zookeeper to ClickHouse keeper. It requires small downtime, but you will have all your zookeeper data inside and your clients will just work when your keeper will be back
- the-alchemist 3y agoIt's been a struggle getting Clickhouse accepted here in the U.S., despite its technical prowess, even prior to the war in Ukraine. I know, I read the blog: > ClickHouse, Inc. is a Delaware company with headquarters in the San Francisco Bay Area. We have no operations in Russia, no Russian investors, and no Russian members of our Board of Directors. We do, however, have an incredibly talented team of Russian software engineers located in Amsterdam, and we could not be more proud to call them colleagues. The FUD is really hard to overcome. This is coming from someone who advocated for Clickhouse, sent some PRs, and did a minor code audit.
- wdb 3y agoNice to see an alternative for Zookeeper that doesn't depend on the Java runtime I thought stuff were supposed to be rewritten Rust /s
- tylerhannan 3y agolol. I was waiting for that somewhere ;)
- deleted 3y ago[deleted]
- mr-karan 3y agoI've been using Clickhouse Keeper in production environment since it's first release and have been really happy. It makes setting up distributed tables in Clickhouse (replicas/shards) quite easy. The only issues I've seen is when inserting at a super high throughput (>1000/sec writes) which is actually more of a n issue with Clickhouse Merge Table tree settings rather than Keeper itself. I've also written about it here: https://mrkaran.dev/posts/clickhouse-replication/ https://mrkaran.dev/posts/clickhouse-replication/
- miljen 3y agoDo you provide a decent C++ client library? ZooKeeper only provides a C library that has certain... disadvantages.
- LispSporks22 3y agoMan i would love to work at one of these companies that let their engineers go off and implement alternatives to widely used and tested open source projects like Zookeeper merely for speculative performance gains and “… C++” edit: I'm not even saying this facetiously, it would be freaking awesome.