4 ms·

jamesblonde's article tried to answer the question: > isn't this just a data warehouse [DW]? My understanding is that the addition of a RowStore to the DW/Col

jamesblonde's article tried to answer the question: > isn't this just a data warehouse [DW]? My understanding is that the addition of a RowStore to the DW/ColumnStore addresses the training phase of Machine Learning. I come from a Data Engineering background. I struggled with the Data Science centric terminology. Uber's 2017 post [1] was helpful in establishing the motivations and terminology of their Michelangelo machine learning (ML) platform. The main distinguishing feature of a Feature Store seems to be that it supports efficient batch downloading of row data that is used as the training dataset. The discussion made more sense once I figured out that feature refers to a column or data field. Figure 4 in the Logical Clocks whitepaper Hopsworks Feature Store [2] helped me understand the architecture better. The architecture appears to be what I would call a Hybrid Transactional/Analytical Processing (HTAP) [3] engine with MySQL cluster acting as the RowStore and Apache Hive acting as the ColumnStore. I'm assuming that the Hopsworks Feature Store periodically merges the MySQL updates into Hive and also provides a mechanism to perform federated queries. The use of Hive seems outdated (vs. say Presto) and I wonder if the use of MySQL is required compared to directly accessing column oriented files like ORC/Parquet/Kudu. [1] https://eng.uber.com/michelangelo-machine-learning-platform/ https://eng.uber.com/michelangelo-machine-learning-platform/ [2] (PDF) https://uploads-ssl.webflow.com/5e6f7cd3ee7f51d539a4da0b/5ef397dce942172c90259858_feature%20management%20platform%20v02.pdf https://uploads-ssl.webflow.com/5e6f7cd3ee7f51d539a4da0b/5ef... [3] https://en.wikipedia.org/wiki/Hybrid_transactional/analytical_processing https://en.wikipedia.org/wiki/Hybrid_transactional/analytica...

jamesblonde 6y ago
Yes, sradman. You describe it correctly here as a HTAP architecture. What we do differently is that the same scaleout consistent metadata layer is used for (1) online FS (NDB - mysql clustrer), (2) offline FS (hive), and (3) HopsFS-S3. That is, the data files for Hive are stored in HopsFS, that in turn stores its data in S3. Because of our metadata architecture, we ensure the consistency of hive data files with hive metadata using foreign keys, and we have extended metadata to store all the feature metadata that is kept consistent with good auld foreign keys and transactions. We also automatically synchronize the metadata to elasticsearch, so you can do free-text search for features, files, extended metadata, etc. So, you can search up to PBs of files/dirs/tables/features/extended-metadata in milliseconds. You can find more info here on the research behind it: https://www.logicalclocks.com/research https://www.logicalclocks.com/research
- sradman 6y ago
  The architecture may be HTAP but the batch read use-case for the RowStore (vs. OLTP) seems to be unique to predictive machine learning (new to me at least). Full Text Search sync is a nice feature as well. Thanks for the thoughtful response and the research link.

jamesblonde 6y ago

Yes, sradman. You describe it correctly here as a HTAP architecture. What we do differently is that the same scaleout consistent metadata layer is used for (1) online FS (NDB - mysql clustrer), (2) offline FS (hive), and (3) HopsFS-S3. That is, the data files for Hive are stored in HopsFS, that in turn stores its data in S3. Because of our metadata architecture, we ensure the consistency of hive data files with hive metadata using foreign keys, and we have extended metadata to store all the feature metadata that is kept consistent with good auld foreign keys and transactions. We also automatically synchronize the metadata to elasticsearch, so you can do free-text search for features, files, extended metadata, etc. So, you can search up to PBs of files/dirs/tables/features/extended-metadata in milliseconds. You can find more info here on the research behind it: https://www.logicalclocks.com/research https://www.logicalclocks.com/research

sradman 6y ago
The architecture may be HTAP but the batch read use-case for the RowStore (vs. OLTP) seems to be unique to predictive machine learning (new to me at least). Full Text Search sync is a nice feature as well. Thanks for the thoughtful response and the research link.