4 ms·
It seems the article suggests piping around and querying tabular data in postgres. I don't see a need for the data lake part. Data Warehouses are used to store
by snidane 6y ago
It seems the article suggests piping around and querying tabular data in postgres. I don't see a need for the data lake part.
Data Warehouses are used to store and process anything which looks like a structured table, or nested tables, so called semi-structured data.
Data lakes are for everything else. Your SQL warehouse can't process or is not ergonomic for processing of:
- purchased data, shared as a zip archive of csv files
- 10 level deep nested json files coming from api calls
- html files from webscrapes
- contents of ftp containing 100s of various csv and other files
- array data used for machine learning algorithms
- pickled python ml models
- yaml configs
- pdf documents such as data dictionaries
- materialized views over raw data
Besides your DWH, you need to have a storage layer, where you store these files and raw data. This is the main reason for why companies have data lake projects. Without some centralised oversight and discipline it only results in a big mess. Note that the centralisation doesn't have to be company-wide, each team or department can maintain their own data lake standard. The more centralised you do it, the more economy of data scale you get, but the harder it gets to enforce and maintain with higher chance of turning into mess again.
I think Data Mesh concept is proposing structuring your org as a bunch of data producing teams, each maintaining their own data lake, instead of having one huge ass lake mess in the middle.
Tools like delta.io and Databricks are giving data lakes full capabilities of a data warehouse, so the difference between data lake and data warehouse is diminishing. These days you can get away without a dedicated DWH and just store everything in a blob store and plug in short-lived processing engines as you wish without vendor lock-ins.