3 ms·

What is your take on the need for time-travel queries for versioned, mutable data? Versioning immutable data items is not enough if you have structured data tha

What is your take on the need for time-travel queries for versioned, mutable data? Versioning immutable data items is not enough if you have structured data that is updated. Every time you update a data item, you store a full copy - not a diff of the actual data. You are not able to make "time-travel queries" - give me the data that was generated in this time-range, for example. For example, if you have a Feature Store for ML, and you want to say "Give me train/test data for these features for the years 2012-2020". This isn't possible with versioned immutable data items. Also, if you don't store the diffs in data - if you store immutable copies, you get explosive growth in data volumes. There are 2 (maybe 3) frameworks that allow such time-travel queries i am aware of: Apache Hudi (Uber) and Databricks Delta. (Apache Iceberg by Netflix will have support soon.) Reference: https://www.logicalclocks.com/blog/mlops-with-a-feature-store https://www.logicalclocks.com/blog/mlops-with-a-feature-stor...

timsehn 7y ago
The storage system we use only stores the rows that change. We have a blog post we're publishing on Wednesday explaining how.
- jamesblonde 7y ago
  That's nice. Do you have any idea if it is possible to translate those rows into higher-level time-travel queries? Like if you could plugin an adapter to transform the rows into a data structure (parquet, arrow, json, whatever) that could be useful to analytics and ML apps?
  - timsehn 7y ago
    Like "as of" queries or history queries? We have both of those. AS OF: https://www.dolthub.com/blog/2020-03-20-querying-historical-data-with-as-of/ https://www.dolthub.com/blog/2020-03-20-querying-historical-... HISTORY SYSTEM TABLE: https://www.dolthub.com/blog/2020-01-23-access-to-everything-through-sql/ https://www.dolthub.com/blog/2020-01-23-access-to-everything... You can run `dolt q -r csv -q <query>` to output whatever you want to a CSV. We would need to do work to output a hierarchical format. I'm sure it's possible to build whatever time travel operation you want. We can produce an audit log of every cell in the database pretty quickly.

timsehn 7y ago

The storage system we use only stores the rows that change. We have a blog post we're publishing on Wednesday explaining how.

jamesblonde 7y ago
That's nice. Do you have any idea if it is possible to translate those rows into higher-level time-travel queries? Like if you could plugin an adapter to transform the rows into a data structure (parquet, arrow, json, whatever) that could be useful to analytics and ML apps?
- timsehn 7y ago
  Like "as of" queries or history queries? We have both of those. AS OF: https://www.dolthub.com/blog/2020-03-20-querying-historical-data-with-as-of/ https://www.dolthub.com/blog/2020-03-20-querying-historical-... HISTORY SYSTEM TABLE: https://www.dolthub.com/blog/2020-01-23-access-to-everything-through-sql/ https://www.dolthub.com/blog/2020-01-23-access-to-everything... You can run `dolt q -r csv -q <query>` to output whatever you want to a CSV. We would need to do work to output a hierarchical format. I'm sure it's possible to build whatever time travel operation you want. We can produce an audit log of every cell in the database pretty quickly.