6 ms·

Author here. I wrote this article because I keep getting the question from prospects - isn't this just a data warehouse? If I missed out on anything or got anyt

Author here. I wrote this article because I keep getting the question from prospects - isn't this just a data warehouse? If I missed out on anything or got anything wrong, please let us know here.

streetcat1 6y ago
Can you please explain how the ONLINE feature store works? I.e. if the prediction request contain, let say, user id, and the user record is not in the ONLINE feature store (e.g. Redis), than you would need to go to the OFFLINE store and do a join, or a select? To sum up, assuming that the online feature store is some sort of a cache, how do you know which objects to place in the cache?
- sidlls 6y ago
  That depends on the organizational structure of the teams, what they’re trying to accomplish, and the budget for resources to name a few things. Generally I don’t view a “feature store” as being a simple lake of raw data. To me a feature store is a place where features known to have signal in a ML model are stored. It has structure, “clean” data (to the extent possible) and some pre-computed elements (e.g. aggregates over some commonly used time windows) to facilitate efficiency in the end to end ML pipeline. I’d also argue a feature store is not complete unless there are tools and infrastructure to make it relatively easy to provision and access new features: that is, it should cover the issues you raise.
  - streetcat1 6y ago
    I am not sure how this answer my question. I understand that a feature store contain the features that have signal. My question is about the operational aspect of online vs offline stores. I.e. what happen if the features needed for a prediction are not in the ONLINE store?
    - skingkong 6y ago
      Yeah in this type of architecture there is some sort of join required. Interestingly, @maycotte talking about the Molecula feature store – it serves real-time and historical features in the same storage layer. Significant performance gains with that approach.
      - jamesblonde 6y ago
        If you want to reuse features, and not write a new pipeline for every model, you need JOINs. Both for the online and offline feature stores.
    - jamesblonde 6y ago
      You can do real-time feature engineering in your application as well. But we recommend to do it in a feature pipeline that also syncs the features to the feature store. We have a talk about it coming out at the spark/ai summit: https://databricks.com/session_eu20/real-time-feature-engineering-with-apache-spark-streaming-and-hof https://databricks.com/session_eu20/real-time-feature-engine...
- jamesblonde 6y ago
  No, the feature would have to be in the online feature store (NDB, not Redis in our case). We have metadata about the schema of the model (the same schema as its training dataset) so that the applications can just supply the IDs and get back the feature vector in the correct order and get it also in the format needed (a npy array or a tfrecord example). This really helps maintenance, as adding a new feature means you don't have to update the application to account for the new feature and ensure the ordering of features matches the ordering in the training dataset.
notsuoh 6y ago
I mean, maybe you're leaving this intentionally open ended to garner comments to get your post higher on the HN page, but perhaps you could answer the question you posted: Isn't this just a data warehouse?
- overfitted 6y ago
  I might be falling for something related to Cunningham's Law here but.. I believe the whole article is an effort of trying to answer this question. Quoting some from the article: "Data warehouses are used primarily by business analysts for interactive querying and for generating historical reports/dashboards on the business. Feature stores are used by both data scientists and by the online/batch applications, and they are fed data by feature pipelines, typically written in Python or Scala/Java. Also, Data warehouses mostly stores data in relational tables, whereas a Feature Store stores it as numerical and categorical features and outputs tensors and/or vectors for training or serving.
- supercanuck 6y ago
  it absolutely is. This company is trying to make a distinction between Online Data (real time streaming with low latency), no joins, key/store and a more traditional batch processing, OLAP type configurations. but modern data warehouses can support both. https://www.snowflake.com/streaming-data/ https://www.snowflake.com/streaming-data/ I think this is an effort to segment the data warehousing market and provide new names for things that already exist and providing a vocabulary to users who may not be familiar with a company's existing datawarehouse solutions
  - jamesblonde 6y ago
    Online data is not necessarily real-time streaming. I am making a distinction between OLTP workloads for the online applications that need a feature vector (i.e., a row of data) to make an individual prediction, and a client that is creating train/test data from millions of rows of data (features) - that is the OLAP workload. To be more concrete, Feast is an open-source Feature Store built on BigQuery and originally BigTable. But the latency of BigTable for the OLTP workload was too high for GoJEK (feature lookup is just one part of making a prediction), so they switched to Redis. Redis PK lookups are a couple of ms, on average, compared with 10+ ms for BigTable. What is the latency of a PK lookup on snowflake? It ain't a millisecond or two. On MySQL Cluster (NDB), our online feature store, PK lookups return in sub-ms latency on dedicated hardware.
sradman 6y ago
jamesblonde's article tried to answer the question: > isn't this just a data warehouse [DW]? My understanding is that the addition of a RowStore to the DW/ColumnStore addresses the training phase of Machine Learning. I come from a Data Engineering background. I struggled with the Data Science centric terminology. Uber's 2017 post [1] was helpful in establishing the motivations and terminology of their Michelangelo machine learning (ML) platform. The main distinguishing feature of a Feature Store seems to be that it supports efficient batch downloading of row data that is used as the training dataset. The discussion made more sense once I figured out that feature refers to a column or data field. Figure 4 in the Logical Clocks whitepaper Hopsworks Feature Store [2] helped me understand the architecture better. The architecture appears to be what I would call a Hybrid Transactional/Analytical Processing (HTAP) [3] engine with MySQL cluster acting as the RowStore and Apache Hive acting as the ColumnStore. I'm assuming that the Hopsworks Feature Store periodically merges the MySQL updates into Hive and also provides a mechanism to perform federated queries. The use of Hive seems outdated (vs. say Presto) and I wonder if the use of MySQL is required compared to directly accessing column oriented files like ORC/Parquet/Kudu. [1] https://eng.uber.com/michelangelo-machine-learning-platform/ https://eng.uber.com/michelangelo-machine-learning-platform/ [2] (PDF) https://uploads-ssl.webflow.com/5e6f7cd3ee7f51d539a4da0b/5ef397dce942172c90259858_feature%20management%20platform%20v02.pdf https://uploads-ssl.webflow.com/5e6f7cd3ee7f51d539a4da0b/5ef... [3] https://en.wikipedia.org/wiki/Hybrid_transactional/analytical_processing https://en.wikipedia.org/wiki/Hybrid_transactional/analytica...
- jamesblonde 6y ago
  Yes, sradman. You describe it correctly here as a HTAP architecture. What we do differently is that the same scaleout consistent metadata layer is used for (1) online FS (NDB - mysql clustrer), (2) offline FS (hive), and (3) HopsFS-S3. That is, the data files for Hive are stored in HopsFS, that in turn stores its data in S3. Because of our metadata architecture, we ensure the consistency of hive data files with hive metadata using foreign keys, and we have extended metadata to store all the feature metadata that is kept consistent with good auld foreign keys and transactions. We also automatically synchronize the metadata to elasticsearch, so you can do free-text search for features, files, extended metadata, etc. So, you can search up to PBs of files/dirs/tables/features/extended-metadata in milliseconds. You can find more info here on the research behind it: https://www.logicalclocks.com/research https://www.logicalclocks.com/research
arethuza 6y ago
I might quibble a bit about: "They could derive historical insights into the business using BI tools." A lot of work in some traditional BI tools (e.g. Oracle Hyperion) is actually about collecting and aggregating forward looking data (within forecasts, budgets, scenarios) that are then compared with actuals.
- jamesblonde 6y ago
  Yes, it was sloppy, sorry. "Derive insights from historical data" would have been more correct.

Author here. I wrote this article because I keep getting the question from prospects - isn't this just a data warehouse? If I missed out on anything or got anything wrong, please let us know here.

streetcat1 6y ago

Can you please explain how the ONLINE feature store works? I.e. if the prediction request contain, let say, user id, and the user record is not in the ONLINE feature store (e.g. Redis), than you would need to go to the OFFLINE store and do a join, or a select? To sum up, assuming that the online feature store is some sort of a cache, how do you know which objects to place in the cache?

sidlls 6y ago
That depends on the organizational structure of the teams, what they’re trying to accomplish, and the budget for resources to name a few things. Generally I don’t view a “feature store” as being a simple lake of raw data. To me a feature store is a place where features known to have signal in a ML model are stored. It has structure, “clean” data (to the extent possible) and some pre-computed elements (e.g. aggregates over some commonly used time windows) to facilitate efficiency in the end to end ML pipeline. I’d also argue a feature store is not complete unless there are tools and infrastructure to make it relatively easy to provision and access new features: that is, it should cover the issues you raise.
- streetcat1 6y ago
  I am not sure how this answer my question. I understand that a feature store contain the features that have signal. My question is about the operational aspect of online vs offline stores. I.e. what happen if the features needed for a prediction are not in the ONLINE store?
  - skingkong 6y ago
    Yeah in this type of architecture there is some sort of join required. Interestingly, @maycotte talking about the Molecula feature store – it serves real-time and historical features in the same storage layer. Significant performance gains with that approach.
    - jamesblonde 6y ago
      If you want to reuse features, and not write a new pipeline for every model, you need JOINs. Both for the online and offline feature stores.
  - jamesblonde 6y ago
    You can do real-time feature engineering in your application as well. But we recommend to do it in a feature pipeline that also syncs the features to the feature store. We have a talk about it coming out at the spark/ai summit: https://databricks.com/session_eu20/real-time-feature-engineering-with-apache-spark-streaming-and-hof https://databricks.com/session_eu20/real-time-feature-engine...
jamesblonde 6y ago
No, the feature would have to be in the online feature store (NDB, not Redis in our case). We have metadata about the schema of the model (the same schema as its training dataset) so that the applications can just supply the IDs and get back the feature vector in the correct order and get it also in the format needed (a npy array or a tfrecord example). This really helps maintenance, as adding a new feature means you don't have to update the application to account for the new feature and ensure the ordering of features matches the ordering in the training dataset.

notsuoh 6y ago

I mean, maybe you're leaving this intentionally open ended to garner comments to get your post higher on the HN page, but perhaps you could answer the question you posted: Isn't this just a data warehouse?

overfitted 6y ago
I might be falling for something related to Cunningham's Law here but.. I believe the whole article is an effort of trying to answer this question. Quoting some from the article: "Data warehouses are used primarily by business analysts for interactive querying and for generating historical reports/dashboards on the business. Feature stores are used by both data scientists and by the online/batch applications, and they are fed data by feature pipelines, typically written in Python or Scala/Java. Also, Data warehouses mostly stores data in relational tables, whereas a Feature Store stores it as numerical and categorical features and outputs tensors and/or vectors for training or serving.
supercanuck 6y ago
it absolutely is. This company is trying to make a distinction between Online Data (real time streaming with low latency), no joins, key/store and a more traditional batch processing, OLAP type configurations. but modern data warehouses can support both. https://www.snowflake.com/streaming-data/ https://www.snowflake.com/streaming-data/ I think this is an effort to segment the data warehousing market and provide new names for things that already exist and providing a vocabulary to users who may not be familiar with a company's existing datawarehouse solutions
- jamesblonde 6y ago
  Online data is not necessarily real-time streaming. I am making a distinction between OLTP workloads for the online applications that need a feature vector (i.e., a row of data) to make an individual prediction, and a client that is creating train/test data from millions of rows of data (features) - that is the OLAP workload. To be more concrete, Feast is an open-source Feature Store built on BigQuery and originally BigTable. But the latency of BigTable for the OLTP workload was too high for GoJEK (feature lookup is just one part of making a prediction), so they switched to Redis. Redis PK lookups are a couple of ms, on average, compared with 10+ ms for BigTable. What is the latency of a PK lookup on snowflake? It ain't a millisecond or two. On MySQL Cluster (NDB), our online feature store, PK lookups return in sub-ms latency on dedicated hardware.

sradman 6y ago

jamesblonde's article tried to answer the question: > isn't this just a data warehouse [DW]? My understanding is that the addition of a RowStore to the DW/ColumnStore addresses the training phase of Machine Learning. I come from a Data Engineering background. I struggled with the Data Science centric terminology. Uber's 2017 post [1] was helpful in establishing the motivations and terminology of their Michelangelo machine learning (ML) platform. The main distinguishing feature of a Feature Store seems to be that it supports efficient batch downloading of row data that is used as the training dataset. The discussion made more sense once I figured out that feature refers to a column or data field. Figure 4 in the Logical Clocks whitepaper Hopsworks Feature Store [2] helped me understand the architecture better. The architecture appears to be what I would call a Hybrid Transactional/Analytical Processing (HTAP) [3] engine with MySQL cluster acting as the RowStore and Apache Hive acting as the ColumnStore. I'm assuming that the Hopsworks Feature Store periodically merges the MySQL updates into Hive and also provides a mechanism to perform federated queries. The use of Hive seems outdated (vs. say Presto) and I wonder if the use of MySQL is required compared to directly accessing column oriented files like ORC/Parquet/Kudu. [1] https://eng.uber.com/michelangelo-machine-learning-platform/ https://eng.uber.com/michelangelo-machine-learning-platform/ [2] (PDF) https://uploads-ssl.webflow.com/5e6f7cd3ee7f51d539a4da0b/5ef397dce942172c90259858_feature%20management%20platform%20v02.pdf https://uploads-ssl.webflow.com/5e6f7cd3ee7f51d539a4da0b/5ef... [3] https://en.wikipedia.org/wiki/Hybrid_transactional/analytical_processing https://en.wikipedia.org/wiki/Hybrid_transactional/analytica...

jamesblonde 6y ago
Yes, sradman. You describe it correctly here as a HTAP architecture. What we do differently is that the same scaleout consistent metadata layer is used for (1) online FS (NDB - mysql clustrer), (2) offline FS (hive), and (3) HopsFS-S3. That is, the data files for Hive are stored in HopsFS, that in turn stores its data in S3. Because of our metadata architecture, we ensure the consistency of hive data files with hive metadata using foreign keys, and we have extended metadata to store all the feature metadata that is kept consistent with good auld foreign keys and transactions. We also automatically synchronize the metadata to elasticsearch, so you can do free-text search for features, files, extended metadata, etc. So, you can search up to PBs of files/dirs/tables/features/extended-metadata in milliseconds. You can find more info here on the research behind it: https://www.logicalclocks.com/research https://www.logicalclocks.com/research

arethuza 6y ago

I might quibble a bit about: "They could derive historical insights into the business using BI tools." A lot of work in some traditional BI tools (e.g. Oracle Hyperion) is actually about collecting and aggregating forward looking data (within forecasts, budgets, scenarios) that are then compared with actuals.

jamesblonde 6y ago
Yes, it was sloppy, sorry. "Derive insights from historical data" would have been more correct.