5 ms·

I mean, maybe you're leaving this intentionally open ended to garner comments to get your post higher on the HN page, but perhaps you could answer the question

I mean, maybe you're leaving this intentionally open ended to garner comments to get your post higher on the HN page, but perhaps you could answer the question you posted: Isn't this just a data warehouse?

overfitted 6y ago
I might be falling for something related to Cunningham's Law here but.. I believe the whole article is an effort of trying to answer this question. Quoting some from the article: "Data warehouses are used primarily by business analysts for interactive querying and for generating historical reports/dashboards on the business. Feature stores are used by both data scientists and by the online/batch applications, and they are fed data by feature pipelines, typically written in Python or Scala/Java. Also, Data warehouses mostly stores data in relational tables, whereas a Feature Store stores it as numerical and categorical features and outputs tensors and/or vectors for training or serving.
supercanuck 6y ago
it absolutely is. This company is trying to make a distinction between Online Data (real time streaming with low latency), no joins, key/store and a more traditional batch processing, OLAP type configurations. but modern data warehouses can support both. https://www.snowflake.com/streaming-data/ https://www.snowflake.com/streaming-data/ I think this is an effort to segment the data warehousing market and provide new names for things that already exist and providing a vocabulary to users who may not be familiar with a company's existing datawarehouse solutions
- jamesblonde 6y ago
  Online data is not necessarily real-time streaming. I am making a distinction between OLTP workloads for the online applications that need a feature vector (i.e., a row of data) to make an individual prediction, and a client that is creating train/test data from millions of rows of data (features) - that is the OLAP workload. To be more concrete, Feast is an open-source Feature Store built on BigQuery and originally BigTable. But the latency of BigTable for the OLTP workload was too high for GoJEK (feature lookup is just one part of making a prediction), so they switched to Redis. Redis PK lookups are a couple of ms, on average, compared with 10+ ms for BigTable. What is the latency of a PK lookup on snowflake? It ain't a millisecond or two. On MySQL Cluster (NDB), our online feature store, PK lookups return in sub-ms latency on dedicated hardware.
- ScoutOrgo 6y ago
  As a data scientist using snowflake and in the market for a feature store, the snowflake streaming is only for data ingestion, not serving. It doesn't solve the problem of serving data for a low latency app.
  - cpdoughe 6y ago
    hi, i'm the co-founder and CTO of a feature store startup that is building this on top of snowflake, can we chat? my name is Patrick, website is rasgoml.com, and e-mail is patrick@rasgoml.com. Thanks!
  - supercanuck 6y ago
    How is that not just a feature of some future Data Warehouse though?
jamesblonde 6y ago
Hi. No, I don't think it is. Because your online applications that need low latency access to features won't tolerate the latencies provided by existing data warehouses. The online app that has an operation model that makes predictions is one client of the feature store. For the other client - a data scientist who is browsing features and creating train/test datasets - yes, that is similar to a data warehouse, except that you get APIs in Python and your data has precomputed statistics that make it better for exploratory data analysis than a traditional data warehouse.
- hodgesrm 6y ago
  Which existing data warehouses do you mean? ClickHouse and Druid can return answers in millisecond. Data warehouses are starting to optimize for low latency response. Disclaimer: My company supports ClickHouse.

overfitted 6y ago

I might be falling for something related to Cunningham's Law here but.. I believe the whole article is an effort of trying to answer this question. Quoting some from the article: "Data warehouses are used primarily by business analysts for interactive querying and for generating historical reports/dashboards on the business. Feature stores are used by both data scientists and by the online/batch applications, and they are fed data by feature pipelines, typically written in Python or Scala/Java. Also, Data warehouses mostly stores data in relational tables, whereas a Feature Store stores it as numerical and categorical features and outputs tensors and/or vectors for training or serving.

supercanuck 6y ago

it absolutely is. This company is trying to make a distinction between Online Data (real time streaming with low latency), no joins, key/store and a more traditional batch processing, OLAP type configurations. but modern data warehouses can support both. https://www.snowflake.com/streaming-data/ https://www.snowflake.com/streaming-data/ I think this is an effort to segment the data warehousing market and provide new names for things that already exist and providing a vocabulary to users who may not be familiar with a company's existing datawarehouse solutions

jamesblonde 6y ago
Online data is not necessarily real-time streaming. I am making a distinction between OLTP workloads for the online applications that need a feature vector (i.e., a row of data) to make an individual prediction, and a client that is creating train/test data from millions of rows of data (features) - that is the OLAP workload. To be more concrete, Feast is an open-source Feature Store built on BigQuery and originally BigTable. But the latency of BigTable for the OLTP workload was too high for GoJEK (feature lookup is just one part of making a prediction), so they switched to Redis. Redis PK lookups are a couple of ms, on average, compared with 10+ ms for BigTable. What is the latency of a PK lookup on snowflake? It ain't a millisecond or two. On MySQL Cluster (NDB), our online feature store, PK lookups return in sub-ms latency on dedicated hardware.
ScoutOrgo 6y ago
As a data scientist using snowflake and in the market for a feature store, the snowflake streaming is only for data ingestion, not serving. It doesn't solve the problem of serving data for a low latency app.
- cpdoughe 6y ago
  hi, i'm the co-founder and CTO of a feature store startup that is building this on top of snowflake, can we chat? my name is Patrick, website is rasgoml.com, and e-mail is patrick@rasgoml.com. Thanks!
- supercanuck 6y ago
  How is that not just a feature of some future Data Warehouse though?

jamesblonde 6y ago

Hi. No, I don't think it is. Because your online applications that need low latency access to features won't tolerate the latencies provided by existing data warehouses. The online app that has an operation model that makes predictions is one client of the feature store. For the other client - a data scientist who is browsing features and creating train/test datasets - yes, that is similar to a data warehouse, except that you get APIs in Python and your data has precomputed statistics that make it better for exploratory data analysis than a traditional data warehouse.

hodgesrm 6y ago
Which existing data warehouses do you mean? ClickHouse and Druid can return answers in millisecond. Data warehouses are starting to optimize for low latency response. Disclaimer: My company supports ClickHouse.