The data science platform looks great, but I have a fundamental issue with building a data versioning platform using git-like semantics - Pachyderm and DVC foll

The data science platform looks great, but I have a fundamental issue with building a data versioning platform using git-like semantics - Pachyderm and DVC follow this approach. Git-like approaches just track immutable files - they do not store the differences between files. That's fine when you're working with small amounts of data, but doesn't scale to handle large volumes of data and prevents you from making some times of time-travel queries - give me the data that changed in this time range. The emerging enterprise approach to handle this is to use a framework like Delta.io, Apache Hudi, or Apache Iceberg. These frameworks add metadata over Parquet, and allow you to do point point-in-time time-travel queries as well as interval-based time-travel queries (give me data that changed in this time interval). Parquet is becoming a dominant file format for data lakes and is cheap to store on s3. Disclosure: I am a co-founder of Logical Clocks.