4 ms·
Date lake can be thought of a file system. Imagine 100 CSVs in folders, usually stored on S3. Data Lakehouse involves adding things like the ability to query
by benjaminwootton 3y ago
Date lake can be thought of a file system. Imagine 100 CSVs in folders, usually stored on S3.
Data Lakehouse involves adding things like the ability to query via SQL, the ability to update/insert/delete, transactions.
Where before people needed warehouses for BI and lakes for data science, they can now have only one approach.
It’s likely to be a big trend as data moves to this format and arrangement and the DBMS vendors like Snowflake have to play nicely with it.
- lelanthran 3y ago> Where before people needed warehouses for BI and lakes for data science, they can now have only one approach. This is all very interesting, and thank you for taking the time to explain. Any good starting points for someone who would like to know more?
- benjaminwootton 3y agoDatabricks popularised the concept and explain it very well - https://youtu.be/g11y-kJHr3I?si=j8FAkFsIjScHv24f https://youtu.be/g11y-kJHr3I?si=j8FAkFsIjScHv24f It’s a technology independent pattern though.
- dmtryshmtv 3y agoSee Databricks authors' paper: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf
- hermitcrab 3y agoData warehouse, data lake, data lakehouse. The data world has some terrible terminology.
- CyberDildonics 3y agoYou can't resell someone a file system if you don't rename it to a "data lake". You can't resell someone an indexed file system from the 80's that can be queried with SQL unless you rename it a "data lakehouse".
- CRConrad 3y agoWait til you dive into ETL vs ELT or even better, if you've been doing ETL since long before "ELT" was "a thing", but everyone did ETL actually in an ELT fashion... It's not only the names, but something like 98% of the tools too, that suck.
- chimerasaurus 3y agoDisclaimer - work at Snowflake. Two quick points to mention. 1. Snowflake has always used blob stores + file data + metadata. Architecturally it’s actually always been very Lakehouse-y 2. Parquet and Iceberg should be equivalent in performance and features. It’s more than playing nicely - it’s more choose your own adventure where all things are equal.