3 ms·

Some high level context for those less familiar with the Lakehouse storage system space. For various reasons, several companies moved from data warehouses to d

Some high level context for those less familiar with the Lakehouse storage system space. For various reasons, several companies moved from data warehouses to data lakes starting around 7-10 years ago. Data lakes are better for ML / AI workloads, cheaper, more flexible, and separate compute from storage. With a data warehouse, you need to share compute with other users. With data lakes you can attach an arbitrary number of computational clusters to the data. Data lakes were limited in many regards. They were easily corrupted (no schema enforcement), required slow file listings when reading data, and didn't support ACID transactions. I'm on the Delta Lake team and will speak to some of the benefits of Delta Lake compared to data lakes: * Delta Lake supports ACID transactions, so Delta tables are harder to corrupt. The transaction log makes it easy to time travel, version datasets, and rollback to earlier versions of your data. * Delta Lake allows for schema enforcement & evolution * Delta Lake makes it easy to compact small files (big data systems don't like an excessive number of small files) * Delta Lake lets readers get files and skip files via the transaction log (much faster than a file listing). Z ORDERING the data makes reads even faster. The Delta Lake protocol is implemented in a Scala library and exposed via PySpark, Scala Spark, and Java Spark bindings. This is the library most people think of when conceptualizing Delta Lake. There is also a Delta Lake Java Standalone library that's used to build other readers like the Trino & Hive readers. The Delta Rust project is another implementation of the Delta Lake protocol that is implemented in Rust. This library is accessible via Rust or Python bindings. Polars just added a Delta Lake reader with delta-rs and this library can also be used to easily read Delta Lakes into other DataFrames like pandas or Dask. Lots of DataFrame users are struggling with data lakes / single data files. They don't have any data skipping capabilities (unless Parquet file footers are read), their datasets are easily corruptible, and they don't have any schema enforcement / schema evolution / data versioning / etc. I expect the data community to accelerate the shift to Lakehouse storage systems as they learn about all of these advantages.

AtlasBarfed 4y ago
Not that I know what anything means in "big data lake OLAP database" anymore, but I always thought a data lake implied a lot of hybrid sources/formats/structures for the data, but the advocacy here implies that the data is all ingested and reformatted, which to me is a data warehouse. But then again, data lake may simply be what a data warehouse is now called in marketspeak. Also, I stopped paying attention when the treadmill of new frameworks became unbearable to track, is spark now settled as the standard of distributed "processing", as in mapreduce / distributed query / distributed batch / etc? I get that performance can improve by unifying to a file format like parquet, but again that seems like a data warehouse. A data lake should be something over heterogenous sources with "drivers" or "adaptors" IMO, in particular because the restoration of the data inputs stays in the knowledge domain of the source production database maintainers.
- MrPowers 4y ago
  You're understandably confused by the industry terminology that's ambiguous and morphing over time. Data lakes are typically CSV/JSON/ORC/Avro/Parquet files stored in a storage system (cloud storage like AWS S3 or HDFS). Data lakes are schema on read (the query engine gets the schema when reading the data). A data warehouse is something like Redshift that bundles storage and compute. You have to buy the storage and compute as a single package. Data warehouses are schema on write. The schema is defined when the table is created. And yes, I'd say that Spark is generally considered the "standard" distributed data processing engine these days although there are alternatives.
vgt 4y ago
You're making a lot of assertions I am not sure I agree with: > Data lakes are better for ML / AI workloads, cheaper, more flexible, and separate compute from storage. With a data warehouse, you need to share compute with other users. With data lakes you can attach an arbitrary number of computational clusters to the data. - I am not sure it's any cheaper than BQ or Snowflake storage. - Modern CDW separates compute from storage. - I am not sure what you mean by "you need to share compute with others". Why? - You can attach an arbitrary number of "clusters" in BQ and Snowflake as well. Additionally, modern CDW provides a very high level of abstraction and a very high level of manageability. Their time travel and compaction actually work, and their storage systems are continuously optimized for optimal performance.
hobs 4y ago
Isn't it ACID transactions on ... one table? I thought joins were right out for that on DLT at this point.

AtlasBarfed 4y ago

Not that I know what anything means in "big data lake OLAP database" anymore, but I always thought a data lake implied a lot of hybrid sources/formats/structures for the data, but the advocacy here implies that the data is all ingested and reformatted, which to me is a data warehouse. But then again, data lake may simply be what a data warehouse is now called in marketspeak. Also, I stopped paying attention when the treadmill of new frameworks became unbearable to track, is spark now settled as the standard of distributed "processing", as in mapreduce / distributed query / distributed batch / etc? I get that performance can improve by unifying to a file format like parquet, but again that seems like a data warehouse. A data lake should be something over heterogenous sources with "drivers" or "adaptors" IMO, in particular because the restoration of the data inputs stays in the knowledge domain of the source production database maintainers.

MrPowers 4y ago
You're understandably confused by the industry terminology that's ambiguous and morphing over time. Data lakes are typically CSV/JSON/ORC/Avro/Parquet files stored in a storage system (cloud storage like AWS S3 or HDFS). Data lakes are schema on read (the query engine gets the schema when reading the data). A data warehouse is something like Redshift that bundles storage and compute. You have to buy the storage and compute as a single package. Data warehouses are schema on write. The schema is defined when the table is created. And yes, I'd say that Spark is generally considered the "standard" distributed data processing engine these days although there are alternatives.

vgt 4y ago

You're making a lot of assertions I am not sure I agree with: > Data lakes are better for ML / AI workloads, cheaper, more flexible, and separate compute from storage. With a data warehouse, you need to share compute with other users. With data lakes you can attach an arbitrary number of computational clusters to the data. - I am not sure it's any cheaper than BQ or Snowflake storage. - Modern CDW separates compute from storage. - I am not sure what you mean by "you need to share compute with others". Why? - You can attach an arbitrary number of "clusters" in BQ and Snowflake as well. Additionally, modern CDW provides a very high level of abstraction and a very high level of manageability. Their time travel and compaction actually work, and their storage systems are continuously optimized for optimal performance.

hobs 4y ago

Isn't it ACID transactions on ... one table? I thought joins were right out for that on DLT at this point.