4 ms·
Iceberg (https://iceberg.apache.org https://iceberg.apache.org) is an open source alternative to Delta Lake that I cannot recommend enough. It organizes your P
by fractaloop 3y ago
Iceberg (https://iceberg.apache.org https://iceberg.apache.org) is an open source alternative to Delta Lake that I cannot recommend enough.
It organizes your Parquet files (or other serialization formats) in a logical structure with snapshots to allow time travel and git-like semantics for data management and Write-Audit-Publish strategies.
My favorite use recently is the idempotent change data capture to ease replication in the event of failures. When your publishing job fails, you can simply replay the same diff between two snapshots and pick up where you left off.
- reactordev 3y agohttps://iceberg.apache.org https://iceberg.apache.org is the correct link.
- Eridrus 3y agoCan you share some references to the git-like semantics? I couldn't find much about merging the branched tables.
- fractaloop 3y agoAFAIK, it’s limited to fast-forward merge strategies, but you can also create or replace branches and tags, along with cherry-picking snapshots. Additional information can be found in: • the branching and tagging DDL (https://iceberg.apache.org/docs/latest/spark-ddl/#branching- https://iceberg.apache.org/docs/latest/spark-ddl/#branching-...) • Iceberg Procedures (https://iceberg.apache.org/docs/latest/spark-procedures/ https://iceberg.apache.org/docs/latest/spark-procedures/)