4 ms·

A small correction about ClickHouse Cloud: 1. It uses a cache on local SSDs ("instance store" in AWS terms) + page cache in memory. 2. This is a write-through

A small correction about ClickHouse Cloud: 1. It uses a cache on local SSDs ("instance store" in AWS terms) + page cache in memory. 2. This is a write-through cache. The data is written into cache and S3. 3. Writes to S3 are synchronous. The write is confirmed only after the data is accepted by S3, and the metadata is accepted by ClickHouse Keeper. 4. The cache is ephemeral (not durable, not redundant, not essential for work). 5. The consistency requirements are fairly strict. Although it is still possible to have a stale read on a replica due to sequentially consistent replication of the metadata, it is not related to S3 or the cache either. 6. It is expected to have low latency on reads, while latency on writes can be tolerated up to around one second. 7. We also tested S3 Express One Zone: https://aws.amazon.com/blogs/storage/clickhouse-cloud-amazon-s3-express-one-zone-making-a-blazing-fast-analytical-database-even-faster/ https://aws.amazon.com/blogs/storage/clickhouse-cloud-amazon...

ec109685 3y ago
With the express one zone architecture, will you move serving to be single AZ so you don’t have to pay cross az charges to read the cached data?
- zX41ZdbW 3y ago
  We've made experiments, but the usage of this tier is still being considered. One of the proposals is: - use it to simplify the cache instead of local SSDs; - it has to be in every AZ where there are replicas; the caches will work independently in each AZ; - so it might be expensive, but if we make replica initialization time close to zero and have no local metadata at all, we can afford to run in one AZ at any moment in time and still have failover to another AZ with the cost of losing the cache. PS. Snowflake has virtual warehouses in a single AZ by default. ClickHouse Cloud has nodes in three AZ (prod tier) or two AZ (dev tier).
jvanlightly 3y ago
Author here: Regarding point 3, it was my understanding that inserts are synchronous but updates may be asynchronous. Is that the case?
- zX41ZdbW 3y ago
  No, updates are also synchronous. A table in ClickHouse is represented by a set of immutable data parts. SELECT query acquires a snapshot of currently active data parts and uses it for reading. INSERT query creates a new data part (or many) and links it to the set. Background merge operations (aka compactions) create new data parts from the existing and remove the existing data parts. The old data parts can still be used in a snapshot by some queries - they are ref-counted and will be removed later by a GC process. Updates are heavy (they are named "mutations"). Mutations create new data parts with modified data and remove the existing data parts. This is similar to merges. There are some optimizations, though: instead of a full copy of the data, a mutation can link the existing data and only add a small patch on top of it, should it be the row numbers of deleted records and the added records or a lazy expression to apply. To summarize: - every data part is immutable; - inserts, selects, merges, and mutations work concurrently with no long locks; - it resembles a lock-free data structure, and we could almost say it is lock-free, but... there are mutexes inside for short locks.

ec109685 3y ago

With the express one zone architecture, will you move serving to be single AZ so you don’t have to pay cross az charges to read the cached data?

zX41ZdbW 3y ago
We've made experiments, but the usage of this tier is still being considered. One of the proposals is: - use it to simplify the cache instead of local SSDs; - it has to be in every AZ where there are replicas; the caches will work independently in each AZ; - so it might be expensive, but if we make replica initialization time close to zero and have no local metadata at all, we can afford to run in one AZ at any moment in time and still have failover to another AZ with the cost of losing the cache. PS. Snowflake has virtual warehouses in a single AZ by default. ClickHouse Cloud has nodes in three AZ (prod tier) or two AZ (dev tier).

jvanlightly 3y ago

Author here: Regarding point 3, it was my understanding that inserts are synchronous but updates may be asynchronous. Is that the case?

zX41ZdbW 3y ago
No, updates are also synchronous. A table in ClickHouse is represented by a set of immutable data parts. SELECT query acquires a snapshot of currently active data parts and uses it for reading. INSERT query creates a new data part (or many) and links it to the set. Background merge operations (aka compactions) create new data parts from the existing and remove the existing data parts. The old data parts can still be used in a snapshot by some queries - they are ref-counted and will be removed later by a GC process. Updates are heavy (they are named "mutations"). Mutations create new data parts with modified data and remove the existing data parts. This is similar to merges. There are some optimizations, though: instead of a full copy of the data, a mutation can link the existing data and only add a small patch on top of it, should it be the row numbers of deleted records and the added records or a lazy expression to apply. To summarize: - every data part is immutable; - inserts, selects, merges, and mutations work concurrently with no long locks; - it resembles a lock-free data structure, and we could almost say it is lock-free, but... there are mutexes inside for short locks.