4 ms·
These seem like a list of common sense practices for any team working with data. It's surprising to see Shopify sticking with the Data Warehouse model in 2020…
by go_prodev 5y ago
These seem like a list of common sense practices for any team working with data.
It's surprising to see Shopify sticking with the Data Warehouse model in 2020… I expected something a little more cutting edge.
Data Warehouses are fine if you are working with a small number of data sources, but at some point they start to slow down development and new analytics, and make it cumbersome for intraday reporting.
This is why I'm seeing Data Warehouses being replaced by Data Lakehouses at my company and others in the industry. The Lakehouse enables faster development and near real-time analytics, on lower cost storage and works with structured and unstructured data. Similar team practices are still applicable, but the underlying data structures and governance is different.
- nojito 5y agoData Lakes are ridiculously expensive. The benefits are almost never worth it these days given how efficient data transformation workflows are now.
- go_prodev 5y agoData Lakes are designed as low cost storage and can be cloud based or on-prem. Not sure what solution you are referring to as being "ridiculously expensive". If the Data is valuable, a data lake should be cost effective.