3 ms·
ClickHouse can work with S3 as a main storage. This is possible because a table is a set of immutable data parts. Data parts can be written once and deleted, po
by zX41ZdbW 3y ago
ClickHouse can work with S3 as a main storage. This is possible because a table is a set of immutable data parts. Data parts can be written once and deleted, possibly as a result of a background merge operation. S3 API is almost enough, except for cases of concurrent database updates. In this case, it is not possible to rely on S3 only because it does not support an atomic "write if not exists" operation. That's why external, strongly consistent metadata storage is needed, which is handled by ClickHouse Keeper.
- afiori 3y agoIs a "write if not exists" atomic operation enouhg as a concurrency primitive for database locks?
- justincormack 3y agoYes, its not necessarily the most efficient mechanism (could be a lot of retries) but its sufficient. See the Delta Lake paper for example [0] [0] https://people.eecs.berkeley.edu/~matei/papers/2020/vldb_delta_lake.pdf https://people.eecs.berkeley.edu/~matei/papers/2020/vldb_del...
- yencabulator 3y agoWhen talking about analytical databases for "big data", yeah. They generally just want a "atomically replace the list of Parquet files that make up this table", with one writer succeeding at a time. That would not be a great base to build a transactional database on.
- mlhpdx 3y agoConditional PUT would be a great addition to S3, indeed.
- buremba 3y agoThat would probably require them to rewrite a non-trivial part of S3 from scratch.
- yencabulator 3y agoGoogle Cloud Storage supports create-if-not-exist and compare-and-swap on generation counter. S3 is much harder to use as a building block without tying your code into a second system like DynamoDB etc. https://pkg.go.dev/cloud.google.com/go/storage#Conditions https://pkg.go.dev/cloud.google.com/go/storage#Conditions