3 ms·
Both Iceberg and Delta Lake support 'time travel' so you can query data as it was at a certain date. SELECT count * FROM my_table AS OF "2025-01-01" https:
by khurs 3mo ago
Both Iceberg and Delta Lake support 'time travel' so you can query data as it was at a certain date.
SELECT count * FROM my_table AS OF "2025-01-01"
https://delta.io/blog/2023-02-01-delta-lake-time-travel/ https://delta.io/blog/2023-02-01-delta-lake-time-travel/
https://iceberg.apache.org/docs/latest/spark-queries/#spark-sql-functions https://iceberg.apache.org/docs/latest/spark-queries/#spark-...
- _zoltan_ 3mo agoup to a limited number of snapshots. it's not arbitrary times, it's when there was an atomic snapshot created in the catalog.
- tomjakubowski 3mo agoDon't know about other systems, but in Ducklake every committed transaction creates a new snapshot. https://ducklake.select/docs/stable/duckdb/usage/snapshots https://ducklake.select/docs/stable/duckdb/usage/snapshots
- _zoltan_ 3mo agobecause they are making an extra step: instead of the snapshot being actually an object manifest, they store the snapshot metadata in a relation database. I have a use case where we do 100k changes a second. I bet you "their unlimited snapshots because we are cool and using dbs for metadata" will fail in such a use case.
- jesseryoung 3mo agoEvery commit in Delta is time travel-able. Defaults in Delta clean up logs in 30 days (and data in 7 days but requires a VACUUM be issued.) You can set those retention periods to any arbitrary time period, but it's typically advised against due to the cost to store all the historical data.