5 ms·

Great article. > data engineers have been increasingly adopting software engineering best practices I think the data engineering field is starting to adopt so

Great article. > data engineers have been increasingly adopting software engineering best practices I think the data engineering field is starting to adopt some software engineering best practices, but it's still really early days. I am the author of popular Spark testing libraries (spark-fast-tests, chispa) and they definitely have a large userbase, but could also grow a lot. > The way organizations structure data teams has changed over the years. Now we see a shift towards decentralized data teams, self-serve data platforms, and ways to store data beyond the data warehouse – such as the data lake, data lakehouse, or the previously mentioned data mesh – to better serve the needs of each data consumer. I think the Lakehouse architecture is the real future of data engineering, see the paper: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf Disclosure: I am on the Delta Lake team, but joined because I believe in the Lakehouse architecture vision. It will take a long time for folks to understand all the differences between data lakes, Lakehouses, data warehouses, etc. Over time, I think mass adoption of the Lakehouse architecture is inevitable (benefits of open file formats, no lock in, separating compute from storage, cost management, scalability, etc.).

victor106 4y ago
> It will take a long time for folks to understand all the differences between data lakes, Lakehouses, data warehouses, etc. What are some good resources that can help educate folks on these differences?
- MrPowers 4y ago
  I am working on some blogs / videos that will hopefully help clarify the differences. I'm working on a Delta Lake vs Parquet blog post right now and gave a 5 Reasons Parquet files are better than CSV talk last year: https://youtu.be/9LYYOdIwQXg https://youtu.be/9LYYOdIwQXg Most of the content that I've seen in this area is really high-level. I'm trying to write posts that are a bit more concrete with some code snippets / high level benchmarks, etc. Hopefully this will help.
- claytonjy 4y ago
  Short version: - data warehouse: schema on write. you have to know the end form before you load it. breaks every time upstream changes (a lot, in this world) - data lake: schema on read. load everything into S3 and deal with it later. Mongo for data platforms - data lakehouse: something in between. store everything loosely like a lake, but have in-lakehouse processes present user-friendly transforms or views like a warehouse. Made possible by cheap storage (parquet on S3), reduces schema breakage by keeping both sides of the T in the same place
  - DougBTX 4y ago
    Materialised views for cloud storage?
- swyx 4y ago
  (OP's coworker) We actually published a guide on data lakes/lakehouses last month! https://airbyte.com/blog/data-lake-lakehouse-guide-powered-by-table-formats-delta-lake-iceberg-hudi https://airbyte.com/blog/data-lake-lakehouse-guide-powered-b... covering: - What’s a Data Lake and Why Do You Need One? - What’s the Differences between a Data Lake, Data Warehouse, and Data Lakehouse - Components of a Data Lake - Data Lake Trends in the Market - How to Turn Your Data Lake into a Lakehouse
- abrazensunset 4y ago
  "Lakehouse" usually means a data lake (bunch of files in object storage with some arbitrary structure) that has an open source "table format" making it act like a database. E.g. using Iceberg or Delta Lake to handle deletes, transactions, concurrency control on top of parquet (the "file format"). The advantage is that various query engines will make it quack like a database, but you have a completely open interop layer that will let any combination of query engines (or just SDKs that implement the table format, or whatever) coexist. And in addition, you can feel good about "owning" your data and not being overtly locked in to Snowflake or Databricks.
chrisjc 4y ago
You mean Lake 'Shanty' Architecture (think DataSwamp vs DataLake) am I right? But in all seriousness, I totally agree with your opinion on LakeHouse Architecture and am especially excited about Apache Iceberg (external table format) and the support and attention it's getting. Although I don't think that selecting any of these data technologies/philosophies comes down to making a mutually exclusive decision. In my opinion, they either build on or compliment each other quite nicely. For those that are interested, here are my descriptions of each... Data Lake Arch - all of your data is stored on blob-storage (S3, etc) in a way that is partitioned thoughtfully and easily accessible, along with a meta-index/catalogue of what data is there, and where it is. Lake House Arch - similar to a DataLake but data is structured and mutable, and hopefully allows for transactions/atomic-ops, schema evolution/drift, time-travel/rollback, so on... Ideally all of the properties that you usually assume to get with any sort of OLAP (maybe even OLTP) DB table. But the most important property in my opinion is that the table is accessible through any compatible compute/query engine/layer. Separating storage and compute has revolutionized the Data Warehouse as we know it, and this is the next iteration of this movement in my opinion. Data Mesh/Grid Arch - designing how the data moves from a source all the way through each and every target while storing/capturing this information in an accessible catalogue/meta-database even as things transpire and change. As a result it provides data lineage and provenance, potentially labeling/tagging, inventory, data-dictionary-like information etc... This one is the most ambiguous and maybe most difficult to describe and probably design/implement, and to be honest I've never seen a real-life working example. I do think this functionality is a critical missing piece of the data stack, whether the solution is a Data Mesh/Grid or something else. Data Engineers have their work cutout on this one, mostly bc this is where their paths cross with those of Application/Service Developers, Software Engineers. In my opinion, developers are usually creating services/applications that are glorified CRUD wrappers around some kind of operational/transactional data store like MySQL, Postgres, Mongo, etc. Analytics, reporting, retention, volume, etc are usually an after thought and not their problem. Until someone hooks the operational data store up to their SQL IDE or Tableau/Looker and takes down prod. Then along comes the data engineer to come up with yet another ETL/ELT to get the data out of the operational data store and into a data warehouse so that reports and analytics can be run without taking prod down again. Data Warehouse (modern) - Massive Parallel Processing (MPP) over detached/separated columnar (for now) data. Some Data Warehouses are already somewhat compatible with Data Lakes since they can use their MPP compute to index and access external tables. Some are already planning to be even more Lake House compatible by not only leveraging their own MPP compute against externally managed tables (eg), but also managing external tables in the first place. That includes managing schemas and running all of the DDLs (CREATE, ALTER, DROP, etc) as well as DQLs (SELECT) and DMLs (MERGE, INSERT, UPDATE, DELETE, ...). Querying data across native DB tables, external tables (potentially from multiple Lake Houses, Data Lakes) all becomes possible with a join in a SQL statement. Additionally this allows for all kinds of governance related functionality as well. Masking, row/column level security, logging, auditing, so on. As you might be able to tell from this post (and my post history) is that I'm a big fan of Snowflake. I'm excited for Snowflake managed Iceberg tables and then consume the data with a different compute/query engine. Snowflake (or other modern DW) could prepare the data (ETL/calc/crunch/etc) and then manage (DDL & DML) it in an Iceberg table. Then something like DuckDB could consume the Iceberg table schema and listen for table changes (oplog?), and then read/query the data performing last-mile analytics (pagination, order, filter, aggs, etc). DuckDB doesn't support Apache Iceberg, but it can read parquet files which are used internally in Iceberg. Obviously supporting external tables is far more complex than just reading a parquet file, but I don't see why this isn't in their future. DuckDB guys, I know you're out there reading this :) https://iceberg.apache.org/ https://iceberg.apache.org/ https://www.snowflake.com/guides/olap-vs-oltp https://www.snowflake.com/guides/olap-vs-oltp https://www.snowflake.com/blog/iceberg-tables-powering-open-standards-with-snowflake-innovations/ https://www.snowflake.com/blog/iceberg-tables-powering-open-... Finally one of my favorite articles: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
oa335 4y ago
I am a data engineer, and I STILL don't understand the differences between the following terms: 1. Data Warehouse 2. Datalake 3. Data Lakehouse 4. Data Mesh Can someone please clearly explain the differences between these concepts?
- antipaul 4y ago
  Marketing and hype. Especially data mesh. I gave up listening to talks/reading papers about it – could never wrap my head around the "mess" Warehouse and Lake also feel mostly like distinctions without difference, though if you really want, I think this is the technical difference: - warehouse: structured data for analytics, outputted from ETL jobs - lake: a dump for unstructured data, to be cleaned up later: EL, then possibly T Moreover, people want to push this even further, with "data river" - streaming(-like) data that is "continuously" transformed (isn't that just an ETL data pipeline?)
- damnruskie 4y ago
  data fabric?
- geoduck14 4y ago
  Let me help you out: Data Warehouse. SQL engine that is much bigger than the databases of yester-year. Their size and scalability give you capabilities that DBAs of a MS SQL dB wouldn't necessarily allow you to do, but think "wow, that is a big database!" Data Lake. Now your database is stored in blob storage (which is like CSV files on your computer) - and the blob storage is HUGE (like hundreds of Terabytes or larger). Also, you store unstructured data, too. So: pictures, JSONs, raw HTML. Whatever Data Lakehouse... do you mean Delta Lake? This is a made up word from Dara Bricks. Take your Data lake, and slap a SQL engine on top of it, and add in some marketing slang ;) Data Mesh. THIS is more about organizations than infrastructure. Imagine a Data Warehouse or Datalake. Now find 5 key stakeholders around your company. Let each of them have a playground in your warehouse. You drop the data into one zone, then they copy it, transform it, and share it in their zone - but they have to play by your rules. Bam. Data Mesh
  - parkerhiggins 4y ago

victor106 4y ago

> It will take a long time for folks to understand all the differences between data lakes, Lakehouses, data warehouses, etc. What are some good resources that can help educate folks on these differences?

MrPowers 4y ago
I am working on some blogs / videos that will hopefully help clarify the differences. I'm working on a Delta Lake vs Parquet blog post right now and gave a 5 Reasons Parquet files are better than CSV talk last year: https://youtu.be/9LYYOdIwQXg https://youtu.be/9LYYOdIwQXg Most of the content that I've seen in this area is really high-level. I'm trying to write posts that are a bit more concrete with some code snippets / high level benchmarks, etc. Hopefully this will help.
claytonjy 4y ago
Short version: - data warehouse: schema on write. you have to know the end form before you load it. breaks every time upstream changes (a lot, in this world) - data lake: schema on read. load everything into S3 and deal with it later. Mongo for data platforms - data lakehouse: something in between. store everything loosely like a lake, but have in-lakehouse processes present user-friendly transforms or views like a warehouse. Made possible by cheap storage (parquet on S3), reduces schema breakage by keeping both sides of the T in the same place
- DougBTX 4y ago
  Materialised views for cloud storage?
swyx 4y ago
(OP's coworker) We actually published a guide on data lakes/lakehouses last month! https://airbyte.com/blog/data-lake-lakehouse-guide-powered-by-table-formats-delta-lake-iceberg-hudi https://airbyte.com/blog/data-lake-lakehouse-guide-powered-b... covering: - What’s a Data Lake and Why Do You Need One? - What’s the Differences between a Data Lake, Data Warehouse, and Data Lakehouse - Components of a Data Lake - Data Lake Trends in the Market - How to Turn Your Data Lake into a Lakehouse
abrazensunset 4y ago
"Lakehouse" usually means a data lake (bunch of files in object storage with some arbitrary structure) that has an open source "table format" making it act like a database. E.g. using Iceberg or Delta Lake to handle deletes, transactions, concurrency control on top of parquet (the "file format"). The advantage is that various query engines will make it quack like a database, but you have a completely open interop layer that will let any combination of query engines (or just SDKs that implement the table format, or whatever) coexist. And in addition, you can feel good about "owning" your data and not being overtly locked in to Snowflake or Databricks.

chrisjc 4y ago

You mean Lake 'Shanty' Architecture (think DataSwamp vs DataLake) am I right? But in all seriousness, I totally agree with your opinion on LakeHouse Architecture and am especially excited about Apache Iceberg (external table format) and the support and attention it's getting. Although I don't think that selecting any of these data technologies/philosophies comes down to making a mutually exclusive decision. In my opinion, they either build on or compliment each other quite nicely. For those that are interested, here are my descriptions of each... Data Lake Arch - all of your data is stored on blob-storage (S3, etc) in a way that is partitioned thoughtfully and easily accessible, along with a meta-index/catalogue of what data is there, and where it is. Lake House Arch - similar to a DataLake but data is structured and mutable, and hopefully allows for transactions/atomic-ops, schema evolution/drift, time-travel/rollback, so on... Ideally all of the properties that you usually assume to get with any sort of OLAP (maybe even OLTP) DB table. But the most important property in my opinion is that the table is accessible through any compatible compute/query engine/layer. Separating storage and compute has revolutionized the Data Warehouse as we know it, and this is the next iteration of this movement in my opinion. Data Mesh/Grid Arch - designing how the data moves from a source all the way through each and every target while storing/capturing this information in an accessible catalogue/meta-database even as things transpire and change. As a result it provides data lineage and provenance, potentially labeling/tagging, inventory, data-dictionary-like information etc... This one is the most ambiguous and maybe most difficult to describe and probably design/implement, and to be honest I've never seen a real-life working example. I do think this functionality is a critical missing piece of the data stack, whether the solution is a Data Mesh/Grid or something else. Data Engineers have their work cutout on this one, mostly bc this is where their paths cross with those of Application/Service Developers, Software Engineers. In my opinion, developers are usually creating services/applications that are glorified CRUD wrappers around some kind of operational/transactional data store like MySQL, Postgres, Mongo, etc. Analytics, reporting, retention, volume, etc are usually an after thought and not their problem. Until someone hooks the operational data store up to their SQL IDE or Tableau/Looker and takes down prod. Then along comes the data engineer to come up with yet another ETL/ELT to get the data out of the operational data store and into a data warehouse so that reports and analytics can be run without taking prod down again. Data Warehouse (modern) - Massive Parallel Processing (MPP) over detached/separated columnar (for now) data. Some Data Warehouses are already somewhat compatible with Data Lakes since they can use their MPP compute to index and access external tables. Some are already planning to be even more Lake House compatible by not only leveraging their own MPP compute against externally managed tables (eg), but also managing external tables in the first place. That includes managing schemas and running all of the DDLs (CREATE, ALTER, DROP, etc) as well as DQLs (SELECT) and DMLs (MERGE, INSERT, UPDATE, DELETE, ...). Querying data across native DB tables, external tables (potentially from multiple Lake Houses, Data Lakes) all becomes possible with a join in a SQL statement. Additionally this allows for all kinds of governance related functionality as well. Masking, row/column level security, logging, auditing, so on. As you might be able to tell from this post (and my post history) is that I'm a big fan of Snowflake. I'm excited for Snowflake managed Iceberg tables and then consume the data with a different compute/query engine. Snowflake (or other modern DW) could prepare the data (ETL/calc/crunch/etc) and then manage (DDL & DML) it in an Iceberg table. Then something like DuckDB could consume the Iceberg table schema and listen for table changes (oplog?), and then read/query the data performing last-mile analytics (pagination, order, filter, aggs, etc). DuckDB doesn't support Apache Iceberg, but it can read parquet files which are used internally in Iceberg. Obviously supporting external tables is far more complex than just reading a parquet file, but I don't see why this isn't in their future. DuckDB guys, I know you're out there reading this :) https://iceberg.apache.org/ https://iceberg.apache.org/ https://www.snowflake.com/guides/olap-vs-oltp https://www.snowflake.com/guides/olap-vs-oltp https://www.snowflake.com/blog/iceberg-tables-powering-open-standards-with-snowflake-innovations/ https://www.snowflake.com/blog/iceberg-tables-powering-open-... Finally one of my favorite articles: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...

oa335 4y ago

I am a data engineer, and I STILL don't understand the differences between the following terms: 1. Data Warehouse 2. Datalake 3. Data Lakehouse 4. Data Mesh Can someone please clearly explain the differences between these concepts?

antipaul 4y ago
Marketing and hype. Especially data mesh. I gave up listening to talks/reading papers about it – could never wrap my head around the "mess" Warehouse and Lake also feel mostly like distinctions without difference, though if you really want, I think this is the technical difference: - warehouse: structured data for analytics, outputted from ETL jobs - lake: a dump for unstructured data, to be cleaned up later: EL, then possibly T Moreover, people want to push this even further, with "data river" - streaming(-like) data that is "continuously" transformed (isn't that just an ETL data pipeline?)
damnruskie 4y ago
data fabric?
geoduck14 4y ago
Let me help you out: Data Warehouse. SQL engine that is much bigger than the databases of yester-year. Their size and scalability give you capabilities that DBAs of a MS SQL dB wouldn't necessarily allow you to do, but think "wow, that is a big database!" Data Lake. Now your database is stored in blob storage (which is like CSV files on your computer) - and the blob storage is HUGE (like hundreds of Terabytes or larger). Also, you store unstructured data, too. So: pictures, JSONs, raw HTML. Whatever Data Lakehouse... do you mean Delta Lake? This is a made up word from Dara Bricks. Take your Data lake, and slap a SQL engine on top of it, and add in some marketing slang ;) Data Mesh. THIS is more about organizations than infrastructure. Imagine a Data Warehouse or Datalake. Now find 5 key stakeholders around your company. Let each of them have a playground in your warehouse. You drop the data into one zone, then they copy it, transform it, and share it in their zone - but they have to play by your rules. Bam. Data Mesh
- parkerhiggins 4y ago