5 ms·

> "ability to time travel for training" Nah, this is nothing new. We've solved this for ages with "snapshots" or "archives", or fancy indexing strategies, or

> "ability to time travel for training" Nah, this is nothing new. We've solved this for ages with "snapshots" or "archives", or fancy indexing strategies, or just a freaking "timestamp" column in your tables.

nikhilsimha 3y ago
Snapshots can’t travel back with milliseconds precision or even minute level precision. They are just full dumps at regular fixed intervals in time.
- _se 3y ago
  Databases have had many forms of time travel for 30+ years now.
  - threeseed 3y ago
    Not at the latency needed for feature serving and most databases struggle with column limits. But please enlighten us on which databases to use so Airbnb (and the rest of us) can stop wasting time.
    - refset 3y ago
      Shameless plug, but XTDB v2 is being built for low-latency bitemporal queries over columnar storage and might be applicable: https://docs.xtdb.com/quickstart/query-the-past.html https://docs.xtdb.com/quickstart/query-the-past.html We've not been developing v2 with ML feature serving in mind so far, but I would love to speak with anyone interested in this use case and figure out where the gaps are.
- mulmen 3y ago
  Snapshots don’t have to be at regular intervals and can be at whatever resolution you choose. You could snapshot as the first step of training then keep that snapshot for the life of the resulting model. Or you could use some other time travel methodology. Snapshots are only one of many options.
  - nikhilsimha 3y ago
    These are reconstruction of features / columns that don’t exist yet.
    - deleted 3y ago
      [deleted]
    - mulmen 3y ago
      I don’t understand what this means. How can something be reconstructed without first existing? Is this not just a caching exercise?
- hobs 3y ago
  https://en.wikipedia.org/wiki/Sixth_normal_form https://en.wikipedia.org/wiki/Sixth_normal_form Basically we've had time travel (via triggers or built in temporal tables or just writing the data) for a long time, its just expensive to have it all for an OLTP database. We've also had slowly changing dimensions to solve this type of problem for a decent amount of time for the labels that sit on top of everything, though really these are just fact tables with a similar historical approach.
  - ezvz 3y ago
    6NF works well for some temporal data, but I haven't seen it work well for windowed aggregations because the start/end time format of saving values doesn't handle events "falling out of the window" too well. At least the examples I've seen have values change due to explicit mutation events.
    - hobs 3y ago
      Agree, you don't really want to pre-aggregate your temporal data, or it will effectively only aggregate at each row-time boundary and the value is lower than just keeping the individual calculations.
ezvz 3y ago
There's a lot more to it than snapshots or timestamped columns when it comes to ML training data generation. We often have windowed aggregations that need to computed as of precise intra-day timestamps in order to achieve parity between training data (backfilled in batch) and the data that is being served online realtime (with streaming aggregations being computed realtime). Standard OLAP solutions right now are really good at "What's the X day sum of this column as of this timestamp", but when every row of your training data has a precise intra-day timestamp that you need windowed aggregations to be accurate as-of, this is a different challenge. And when you have many people sharing these aggregations, but with potentially different timestamps/timelines, you also want them sharing partial aggregations where possibly for efficiency. All of this is well beyond the scope that is addressed by standard OLAP data solutions. Not to mention the fact that the offline computation needs to translate seamlessly to power online serving (i.e. seeding feature values, and combining with streaming realtime aggregations), and the need for online/offline consistency measurement. That's why a lot of teams don't even bother with this, and basically just log their feature values from online to offline. But this limits what kind of data they can use, and also how quickly they can iterate on new features (need to wait for enough log data to accumulate before you can train).
- mulmen 3y ago
  I’m still not seeing how this is a novel problem. You just apply a filter to your timestamp column and re-run the window function. It will give you the same value down to the resolution of the timestamp every time.
  - ezvz 3y ago
    Let's try an example: `average page views in the last 1, 7, 30, 60, 180 days` You need these values accurate as of ~500k timestamps for 10k different page ids, with significant skew for some page ids. So you have a "left" table with 500k rows, each with a page id and timestamp. Then you have a `page_views` table with many millions/billions/whatever rows that need to be aggregated. Sure, you could do this with backfill with SQL and fancy window functions. But let's just look at what you would need to do to actually make this work, assuming you wanted it to be serving online with realtime updates (from a page_views kafka topic that is the source of the page views table): For online serving: 1. Decompose the batch computation to SUM and COUNT and seed the values in your KV store 2. Write the streaming job that does realtime updates to your SUMs/COUNTs. 3. Have an API for fetching and finalizing the AVERAGE value. For Backfilling: 1. Write your verbose query with windowed aggregations (I encourage you to actually try it). 2. Often you also want a daily front-fill job for scheduled retraining. Now you're also thinking about how to reuse previous values. Maybe you reuse your decomposed SUMs/COUNTs above, but if so you're now orchestrating these pipelines. For making sure you didn't mess it up: 1. Compare logs of fetched features to backfilled values to make sure that they're temporally consistent. For sharing: 1. Let's say other ML practitioners are also playing around with this feature, but with a different timelines (i.e. different timestamps). Are they redoing all of the computation? Or are you orchestrating caching and reusing partial windows? So you can do all that, or you can write a few lines of python in Chronon. Now let's say you want to add a window. Or say you want to change it so it's aggregated by `user_id` rather than `page_id`. Or say you want to add other aggregations other than AVERAGE. You can redo all of that again, or change a few lines of Python.
echrisinger 3y ago
What's with the dismissiveness? The author is a senior staff engineer at a huge company & has worked in this space for years. I'd suspect they've done their diligence...
- deleted 3y ago
  [deleted]

nikhilsimha 3y ago

Snapshots can’t travel back with milliseconds precision or even minute level precision. They are just full dumps at regular fixed intervals in time.

_se 3y ago
Databases have had many forms of time travel for 30+ years now.
- threeseed 3y ago
  Not at the latency needed for feature serving and most databases struggle with column limits. But please enlighten us on which databases to use so Airbnb (and the rest of us) can stop wasting time.
  - refset 3y ago
    Shameless plug, but XTDB v2 is being built for low-latency bitemporal queries over columnar storage and might be applicable: https://docs.xtdb.com/quickstart/query-the-past.html https://docs.xtdb.com/quickstart/query-the-past.html We've not been developing v2 with ML feature serving in mind so far, but I would love to speak with anyone interested in this use case and figure out where the gaps are.
mulmen 3y ago
Snapshots don’t have to be at regular intervals and can be at whatever resolution you choose. You could snapshot as the first step of training then keep that snapshot for the life of the resulting model. Or you could use some other time travel methodology. Snapshots are only one of many options.
- nikhilsimha 3y ago
  These are reconstruction of features / columns that don’t exist yet.
  - deleted 3y ago
    [deleted]
  - mulmen 3y ago
    I don’t understand what this means. How can something be reconstructed without first existing? Is this not just a caching exercise?
hobs 3y ago
https://en.wikipedia.org/wiki/Sixth_normal_form https://en.wikipedia.org/wiki/Sixth_normal_form Basically we've had time travel (via triggers or built in temporal tables or just writing the data) for a long time, its just expensive to have it all for an OLTP database. We've also had slowly changing dimensions to solve this type of problem for a decent amount of time for the labels that sit on top of everything, though really these are just fact tables with a similar historical approach.
- ezvz 3y ago
  6NF works well for some temporal data, but I haven't seen it work well for windowed aggregations because the start/end time format of saving values doesn't handle events "falling out of the window" too well. At least the examples I've seen have values change due to explicit mutation events.
  - hobs 3y ago
    Agree, you don't really want to pre-aggregate your temporal data, or it will effectively only aggregate at each row-time boundary and the value is lower than just keeping the individual calculations.

ezvz 3y ago

There's a lot more to it than snapshots or timestamped columns when it comes to ML training data generation. We often have windowed aggregations that need to computed as of precise intra-day timestamps in order to achieve parity between training data (backfilled in batch) and the data that is being served online realtime (with streaming aggregations being computed realtime). Standard OLAP solutions right now are really good at "What's the X day sum of this column as of this timestamp", but when every row of your training data has a precise intra-day timestamp that you need windowed aggregations to be accurate as-of, this is a different challenge. And when you have many people sharing these aggregations, but with potentially different timestamps/timelines, you also want them sharing partial aggregations where possibly for efficiency. All of this is well beyond the scope that is addressed by standard OLAP data solutions. Not to mention the fact that the offline computation needs to translate seamlessly to power online serving (i.e. seeding feature values, and combining with streaming realtime aggregations), and the need for online/offline consistency measurement. That's why a lot of teams don't even bother with this, and basically just log their feature values from online to offline. But this limits what kind of data they can use, and also how quickly they can iterate on new features (need to wait for enough log data to accumulate before you can train).

mulmen 3y ago
I’m still not seeing how this is a novel problem. You just apply a filter to your timestamp column and re-run the window function. It will give you the same value down to the resolution of the timestamp every time.
- ezvz 3y ago
  Let's try an example: `average page views in the last 1, 7, 30, 60, 180 days` You need these values accurate as of ~500k timestamps for 10k different page ids, with significant skew for some page ids. So you have a "left" table with 500k rows, each with a page id and timestamp. Then you have a `page_views` table with many millions/billions/whatever rows that need to be aggregated. Sure, you could do this with backfill with SQL and fancy window functions. But let's just look at what you would need to do to actually make this work, assuming you wanted it to be serving online with realtime updates (from a page_views kafka topic that is the source of the page views table): For online serving: 1. Decompose the batch computation to SUM and COUNT and seed the values in your KV store 2. Write the streaming job that does realtime updates to your SUMs/COUNTs. 3. Have an API for fetching and finalizing the AVERAGE value. For Backfilling: 1. Write your verbose query with windowed aggregations (I encourage you to actually try it). 2. Often you also want a daily front-fill job for scheduled retraining. Now you're also thinking about how to reuse previous values. Maybe you reuse your decomposed SUMs/COUNTs above, but if so you're now orchestrating these pipelines. For making sure you didn't mess it up: 1. Compare logs of fetched features to backfilled values to make sure that they're temporally consistent. For sharing: 1. Let's say other ML practitioners are also playing around with this feature, but with a different timelines (i.e. different timestamps). Are they redoing all of the computation? Or are you orchestrating caching and reusing partial windows? So you can do all that, or you can write a few lines of python in Chronon. Now let's say you want to add a window. Or say you want to change it so it's aggregated by `user_id` rather than `page_id`. Or say you want to add other aggregations other than AVERAGE. You can redo all of that again, or change a few lines of Python.

echrisinger 3y ago

What's with the dismissiveness? The author is a senior staff engineer at a huge company & has worked in this space for years. I'd suspect they've done their diligence...

deleted 3y ago
[deleted]