3 ms·
Iceberg was acquired for $1Bln to create a similar standard called Iceberg. It's worth highlighting the pros/cons of this approach. Pros: + straightforward,
by jakozaur 2y ago
Iceberg was acquired for $1Bln to create a similar standard called Iceberg.
It's worth highlighting the pros/cons of this approach.
Pros:
+ straightforward, no running metadata server needed, scales to zero compute
+ zero-copy clones, time travel/versioning and sharable. Git style, but for your data
+ can work with many vendors without the need to copy data
Cons:
- single optimistic thread lock. This does not work if you have many clients doing many small writes concurrently. You need to batch write, ideally in an append-mainly way.
- quite low-level. You still need someone doing compaction, access control, etc.
- you miss some database features (e.g. skip index based on bloomfilter), some other such as materialized views based on data will be hard to implement if you directly write to blob storage
The data lake is amazing for cold data; for fast-changing recent data (e.g. last 1 day), it is better to use different data storage.