3 ms·

> What if you need to recompute this feature daily, for millions of customers and orders?...moreover, the DS/ML folks want the same feature but for "every 1/3/7

> What if you need to recompute this feature daily, for millions of customers and orders?...moreover, the DS/ML folks want the same feature but for "every 1/3/7/30/90/180/365/730/etc." days breakdown, as well as a bunch of variations about orders and things other than orders (e.g. "average time between new orders, over past 7/30/90 days" or "average $ spent over past 7/30/90 days", as features that serves as a proxy for frequency of activity and level of engagement) ...what data warehouse/database are you using that can't support those queries on-demand at low-latency and granularity? It seems to me that these 'feature stores' are just tables of pre-computed common aggregates, optionally with some kind of versioning id/timestamp. In which case, why do I need some special service to do this, can't I just make a second database in my warehouse and write something that just computes all desired aggregations periodically and dumps them into the second db? Even easier, just use stored process and distribute them to data scientists, and use timestamp parameters to control versioning.

strgcmc 6y ago
You can certainly get pretty far with that approach, and I wouldn't suggest doing anything else until you've outgrown it. Semi-/un-related anecdote, but looking back at what my team was like 6 years ago, we used to be a classic enterprise data warehousing team, and our worldview at the time was basically that, "everything is just a bunch of databases or file systems, and our job is to copy (literally scp) files from source A to destination B." All problems (literally, all) would be solved as one of, "how can I export a file from this source, and what kind of table do we need to create in the DWH?" By that logic, we would make decisions like, "why do you need Splunk/Promethus/Datadog/etc. for operational monitoring, when you could just write those metrics to tables in the DWH?". In retrospect, what is surprising about such a worldview, is not that it was so limiting or that it hurt us in the long run (it did, but less than one might expect), but how far you could actually go and how much you could actually achieve, with such simple primitives.

strgcmc 6y ago

You can certainly get pretty far with that approach, and I wouldn't suggest doing anything else until you've outgrown it. Semi-/un-related anecdote, but looking back at what my team was like 6 years ago, we used to be a classic enterprise data warehousing team, and our worldview at the time was basically that, "everything is just a bunch of databases or file systems, and our job is to copy (literally scp) files from source A to destination B." All problems (literally, all) would be solved as one of, "how can I export a file from this source, and what kind of table do we need to create in the DWH?" By that logic, we would make decisions like, "why do you need Splunk/Promethus/Datadog/etc. for operational monitoring, when you could just write those metrics to tables in the DWH?". In retrospect, what is surprising about such a worldview, is not that it was so limiting or that it hurt us in the long run (it did, but less than one might expect), but how far you could actually go and how much you could actually achieve, with such simple primitives.