4 ms·

I have dvc pipelines such that input/output is iceberg snapshot files. The data gets medium-big and it works well.

• nerdponx 4y ago
  I never heard of Apache Iceberg before. I've used Databricks Delta Lake; is it similar? What is a snapshot file in this case?
  • henrydark 4y ago
    It's basically the same, I just went with iceberg because the specification is a bit more transparent
    • nerdponx 4y ago
      Interesting. So the snapshot file acts much in the same way as a manual "sentinel" file? I generally try to avoid such things because they are brittle and it's easy make a mistake and get the "ad hoc database on your filesystem" out of sync with the actual data.