3 ms·

Thanks for sharing! If anyone has any questions, I'll do my best to get them answered. (Disclaimer: I work at ClickHouse)

deleted 3y ago
[deleted]
alchemist1e9 3y ago
Is there a python client library you can recommend?
- alesapin 3y ago
  All ZooKeeper libraries are compatible with clickhouse-keeper. The most popular and mature is https://kazoo.readthedocs.io/en/latest/ https://kazoo.readthedocs.io/en/latest/. We use it in our integration tests framework (with clickhouse-keeper) a lot.
- zX41ZdbW 3y ago
  The same library that you use for ZooKeeper - kazoo. Note: our stress tests have found a segmentation fault in Python's kazoo library. We only wanted to test Keeper, but found every bug around it :) Let me find a link.
  - zX41ZdbW 3y ago
    https://github.com/ClickHouse/ClickHouse/issues/45367 https://github.com/ClickHouse/ClickHouse/issues/45367
    - qoega 3y ago
      Did not expect to see issue I created
secondcoming 3y ago
What do you use for network stuff in C++, ASIO?
- alesapin 3y ago
  Yes, for internal RAFT implementation boost.asio is used.
pdeva1 3y ago
1. can this be used without clickhouse as just a zookeeper replacement? 2. am i correct in that its using s3 as disk? so can it be run as stateless pods in k8s? 3. if it uses s3, how are latency and costs of PUTs affected? does every write result in a PUT call to s3?
- alesapin 3y ago
  1. Absolutely. clickhouse-keeper is distributed as a standalone static binary or .deb package or .rpm package. You can use it without clickhouse as ZooKeeper replacement. 2. It's not recommended to use slow storage devices for logs in any coordination system (zookeeper, clickhouse-keeper, etcd and so on). Good setup will be small fast SSD/EBS disk for fresh logs and old logs + snapshots offloaded to S3. In such setup the amount of PUT requests will be tiny and latency will be as good as possible.
- pradeepchhetri 3y ago
  Sometime back, I tried using clickhouse-keeper as zookeeper alternative with few other systems like kafka, mesos, solr, Wrote some notes here: https://pradeepchhetri.xyz/clickhousekeeper/ https://pradeepchhetri.xyz/clickhousekeeper/
- zX41ZdbW 3y ago
  1. Yes, it can be used with other applications as a ZooKeeper replacement, unless some unusual ZooKeper features are used (there is no Kerberos integration in Keeper, and it does not support the TTL of persistent nodes) or the application tests for a specific ZooKeeper version. 2. It could be configure to store - snapshots; - RAFT logs other than the latest log; in S3. It cannot use a stateless Kubernetes pod - the latest log has to be located on the filesystem. Although I see you can make a multi-region setup with multiple independent Kubernetes clusters and store logs in tmpfs (which is not 100% wrong from a theoretical standpoint), it is too risky to be practical. 3. Only the snapshots and the previous logs could be on S3, so the PUT requests are done only on log rotation.
  - pdeva1 3y ago
    2. ok. so can i rebuild a cluster with just state in s3? eg: i create a cluster with local disks and s3 backing. entire cluster gets deleted. if i recreate cluster and point to same s3 bucket, will it restore its state?
abronan 3y ago
Thanks for this excellent article! Enjoyed it from start to finish. This gave me a good memory of the work we've done at docker embedding our own replicated and consistent metadata storage using etcd's raft library. Looking at the initial pull request, is it correct that ClickHouse Keeper is based on Ebay's NuRaft library? Or did the Clickhouse team fork and modified this library to accommodate for ClickHouse usage and performance needs?
- alesapin 3y ago
  Yes, you are right ClickHouse Keeper is based on NuRaft. We did a lot of modifications for this library, both for correctness and performance. Almost all of them (need to check) are contributed back to upstream ebay/NuRaft library.

deleted 3y ago

[deleted]

alchemist1e9 3y ago

Is there a python client library you can recommend?

alesapin 3y ago
All ZooKeeper libraries are compatible with clickhouse-keeper. The most popular and mature is https://kazoo.readthedocs.io/en/latest/ https://kazoo.readthedocs.io/en/latest/. We use it in our integration tests framework (with clickhouse-keeper) a lot.
zX41ZdbW 3y ago
The same library that you use for ZooKeeper - kazoo. Note: our stress tests have found a segmentation fault in Python's kazoo library. We only wanted to test Keeper, but found every bug around it :) Let me find a link.
- zX41ZdbW 3y ago
  https://github.com/ClickHouse/ClickHouse/issues/45367 https://github.com/ClickHouse/ClickHouse/issues/45367
  - qoega 3y ago
    Did not expect to see issue I created

secondcoming 3y ago

What do you use for network stuff in C++, ASIO?

alesapin 3y ago
Yes, for internal RAFT implementation boost.asio is used.

pdeva1 3y ago

1. can this be used without clickhouse as just a zookeeper replacement? 2. am i correct in that its using s3 as disk? so can it be run as stateless pods in k8s? 3. if it uses s3, how are latency and costs of PUTs affected? does every write result in a PUT call to s3?

alesapin 3y ago
1. Absolutely. clickhouse-keeper is distributed as a standalone static binary or .deb package or .rpm package. You can use it without clickhouse as ZooKeeper replacement. 2. It's not recommended to use slow storage devices for logs in any coordination system (zookeeper, clickhouse-keeper, etcd and so on). Good setup will be small fast SSD/EBS disk for fresh logs and old logs + snapshots offloaded to S3. In such setup the amount of PUT requests will be tiny and latency will be as good as possible.
pradeepchhetri 3y ago
Sometime back, I tried using clickhouse-keeper as zookeeper alternative with few other systems like kafka, mesos, solr, Wrote some notes here: https://pradeepchhetri.xyz/clickhousekeeper/ https://pradeepchhetri.xyz/clickhousekeeper/
zX41ZdbW 3y ago
1. Yes, it can be used with other applications as a ZooKeeper replacement, unless some unusual ZooKeper features are used (there is no Kerberos integration in Keeper, and it does not support the TTL of persistent nodes) or the application tests for a specific ZooKeeper version. 2. It could be configure to store - snapshots; - RAFT logs other than the latest log; in S3. It cannot use a stateless Kubernetes pod - the latest log has to be located on the filesystem. Although I see you can make a multi-region setup with multiple independent Kubernetes clusters and store logs in tmpfs (which is not 100% wrong from a theoretical standpoint), it is too risky to be practical. 3. Only the snapshots and the previous logs could be on S3, so the PUT requests are done only on log rotation.
- pdeva1 3y ago
  2. ok. so can i rebuild a cluster with just state in s3? eg: i create a cluster with local disks and s3 backing. entire cluster gets deleted. if i recreate cluster and point to same s3 bucket, will it restore its state?

abronan 3y ago

Thanks for this excellent article! Enjoyed it from start to finish. This gave me a good memory of the work we've done at docker embedding our own replicated and consistent metadata storage using etcd's raft library. Looking at the initial pull request, is it correct that ClickHouse Keeper is based on Ebay's NuRaft library? Or did the Clickhouse team fork and modified this library to accommodate for ClickHouse usage and performance needs?

alesapin 3y ago
Yes, you are right ClickHouse Keeper is based on NuRaft. We did a lot of modifications for this library, both for correctness and performance. Almost all of them (need to check) are contributed back to upstream ebay/NuRaft library.