5 ms·

I can't speak highly enough of Clickhouse. Since 20.x, which was already one of (the?) fastest column store databases around, it's gotten even better: - Built

by tonyhb 5y ago

I can't speak highly enough of Clickhouse. Since 20.x, which was already one of (the?) fastest column store databases around, it's gotten even better: - Built-in replication without the need to run Zookeeper, with multi-tier sharding support. They rewrite the ZK protocol using multi-paxos within CH itself. It's great. - Built-in batch inserts via the HTTP protocol... sorry, TCP :(. Previously you'd have to batch using buffer tables, proxies, or in-memory buffering within your client apps. This is no longer needed! - Better support for external data formats (avro, parquet) It's just... so good.

karterk 5y ago
Do you have to do anything special to opt-in for the built-in batch inserts? Earlier, I was forced to use the buffer tables approach: how would one ditch that now?
- tonyhb 5y ago
  If you're using the HTTP protocol, add `async_insert=1` to your connection string. You can tune the batching here: https://clickhouse.com/docs/en/operations/settings/settings/#async-insert https://clickhouse.com/docs/en/operations/settings/settings/...
MrBuddyCasino 5y ago
> Built-in batch inserts via the HTTP protocol Very welcome. We used to do that with a dedicated app. > It's just... so good. What Postgres is for RDBMs and SqlLite for embedded, ClickHouse is for time series. Tastefully designed and driven by engineering excellence. I wish them all the best.
dunkelheit 5y ago
> Built-in replication without the need to run Zookeeper, with multi-tier sharding support. They rewrite the ZK protocol using multi-paxos within CH itself. It's great. AFAIK this description is kind of misleading. When they say that they got rid of Zookeeper people expect that they can just connect clickhouse nodes to each other and the replication will work. But that is not how things work - you still have to run external service called clickhouse-keeper. Basically what they did is they rewrote Zookeeper in C++.
- mr-karan 5y ago
  Fwiw, Clickhouse Keeper _can_ be run as an external daemon if you wish so. But it's packaged with the server binary itself, so once you add <keeper_config> to all your server nodes, you're good to go, without running anything else.
- tonyhb 5y ago
  You can run clickhouse-keeper embedded in the server, though. That way, each "primary" handles incoming SQL connections _and_ paxos communication without additional infra.
ahachete 5y ago
> They rewrite the ZK protocol using multi-paxos within CH itself. It appears to be implemented with Raft, not Paxos (per https://presentations.clickhouse.com/meetup54/keeper.pdf https://presentations.clickhouse.com/meetup54/keeper.pdf, slide 21).
- ahachete 5y ago
  Please also note that there are two concepts here that can easily get mixed. For clarification: * Zookeeper's wire protocol is emulated in CH-Keeper. Nice! So all clients are compatible, etc. * Zookeeper uses a distributed consensus algorithm called ZAB. Which is not Paxos --but many believes so. CH-Keeper uses Raft, and it can do so as the consensus algorithm is not exposed directly: it is an internal property hidden behind the API and obviously the wire protocol.

by tonyhb 5y ago

karterk 5y ago

Do you have to do anything special to opt-in for the built-in batch inserts? Earlier, I was forced to use the buffer tables approach: how would one ditch that now?

tonyhb 5y ago
If you're using the HTTP protocol, add `async_insert=1` to your connection string. You can tune the batching here: https://clickhouse.com/docs/en/operations/settings/settings/#async-insert https://clickhouse.com/docs/en/operations/settings/settings/...

MrBuddyCasino 5y ago

> Built-in batch inserts via the HTTP protocol Very welcome. We used to do that with a dedicated app. > It's just... so good. What Postgres is for RDBMs and SqlLite for embedded, ClickHouse is for time series. Tastefully designed and driven by engineering excellence. I wish them all the best.

dunkelheit 5y ago

> Built-in replication without the need to run Zookeeper, with multi-tier sharding support. They rewrite the ZK protocol using multi-paxos within CH itself. It's great. AFAIK this description is kind of misleading. When they say that they got rid of Zookeeper people expect that they can just connect clickhouse nodes to each other and the replication will work. But that is not how things work - you still have to run external service called clickhouse-keeper. Basically what they did is they rewrote Zookeeper in C++.

mr-karan 5y ago
Fwiw, Clickhouse Keeper _can_ be run as an external daemon if you wish so. But it's packaged with the server binary itself, so once you add <keeper_config> to all your server nodes, you're good to go, without running anything else.
tonyhb 5y ago
You can run clickhouse-keeper embedded in the server, though. That way, each "primary" handles incoming SQL connections _and_ paxos communication without additional infra.

ahachete 5y ago

> They rewrite the ZK protocol using multi-paxos within CH itself. It appears to be implemented with Raft, not Paxos (per https://presentations.clickhouse.com/meetup54/keeper.pdf https://presentations.clickhouse.com/meetup54/keeper.pdf, slide 21).

ahachete 5y ago
Please also note that there are two concepts here that can easily get mixed. For clarification: * Zookeeper's wire protocol is emulated in CH-Keeper. Nice! So all clients are compatible, etc. * Zookeeper uses a distributed consensus algorithm called ZAB. Which is not Paxos --but many believes so. CH-Keeper uses Raft, and it can do so as the consensus algorithm is not exposed directly: it is an internal property hidden behind the API and obviously the wire protocol.