4 ms·
That depends on the organizational structure of the teams, what they’re trying to accomplish, and the budget for resources to name a few things. Generally I do
by sidlls 6y ago
That depends on the organizational structure of the teams, what they’re trying to accomplish, and the budget for resources to name a few things.
Generally I don’t view a “feature store” as being a simple lake of raw data. To me a feature store is a place where features known to have signal in a ML model are stored. It has structure, “clean” data (to the extent possible) and some pre-computed elements (e.g. aggregates over some commonly used time windows) to facilitate efficiency in the end to end ML pipeline.
I’d also argue a feature store is not complete unless there are tools and infrastructure to make it relatively easy to provision and access new features: that is, it should cover the issues you raise.
- streetcat1 6y agoI am not sure how this answer my question. I understand that a feature store contain the features that have signal. My question is about the operational aspect of online vs offline stores. I.e. what happen if the features needed for a prediction are not in the ONLINE store?
- skingkong 6y agoYeah in this type of architecture there is some sort of join required. Interestingly, @maycotte talking about the Molecula feature store – it serves real-time and historical features in the same storage layer. Significant performance gains with that approach.
- jamesblonde 6y agoIf you want to reuse features, and not write a new pipeline for every model, you need JOINs. Both for the online and offline feature stores.
- jamesblonde 6y agoYou can do real-time feature engineering in your application as well. But we recommend to do it in a feature pipeline that also syncs the features to the feature store. We have a talk about it coming out at the spark/ai summit: https://databricks.com/session_eu20/real-time-feature-engineering-with-apache-spark-streaming-and-hof https://databricks.com/session_eu20/real-time-feature-engine...