5 ms·
Delta Lake vs. Parquet: A Comparison
- MrPowers 3y agoData Lakes (i.e. Parquet files in storage without a metadata layer) don't support transactions, require expensive file listing operations, and don't support basic DML operations like deleting rows. Delta Lake stores data in Parquet files and adds a metadata layer to provide support for ACID transactions, schema enforcement, versioned data, and full DML support. Delta Lake also offers concurrency protection. This post explains all the features offered by Delta Lake in comparison to a plain vanilla Parquet data lake.
- alexmolas 3y agoPlease, stop using LLM to provide post summaries. This comment is not adding value to the conversation.
- MrPowers 3y agoI actually wrote this. I thought it was going to be part of the post description and didn't realize it was going to be a comment.
- alexmolas 3y agoSorry if my comment sound too harsh. I've noticed a lot of people commenting autogenerated summaries of the posts, trying to farm karma I guess.
- Zizizizz 3y agoDelta is pretty great, let's you do upserts into tables in DataBricks much easier than without it. I think the website is here: https://delta.io https://delta.io
- MrPowers 3y agoYea, there is a Rust implementation of the Delta Lake protocol that lets you do upserts without Spark too. This allows pandas, Polars, DataFusion, and PyArrow users to easily do upserts as well.
- orthoxerox 3y agoDelta is nice, but a lot of features are missing from the FOSS version. Hudi is nice, but they are in the middle of a big format change right now. Iceberg is nice, but is the most conservative and slow format out of three.
- MrPowers 3y agoThere are a few features missing from the FOSS Scala/Spark implementation of Delta Lake, but I wouldn't say a lot. The FOSS version supports all the table features in the Delta Lake protocol. The Delta Rust implementation is missing more table features, but we're closing the gap fast. We just added support for constraints to Delta Rust and are working on change data feed right now.
- orthoxerox 3y agoDelta Live Tables and automatic vacuuming are the two big features I'm missing.
- chimerasaurus 3y agoI’d take issue with the “Iceberg is slow” theme that Databricks in particular has tried to push. If that were true, Snowflake would not be as fast on Iceberg/Parquet as its native format. The engine makes something fast or slow, not the table format. Disclaimer - am at Snowflake.
- 3y ago
- querez 3y agoI'm not well versed in these things, but at this point, aren't you re-inventing database systems? Talking about things like ACID transactions, schema evolution, dropping columns, ... in the context of a file-format feels bizarre to me.
- ed_elliott_asc 3y agoYep, that is the goal to make the files in your data like queryable like a database.
- gmt2027 3y agoThis is exactly right. The lakehouse is a custom data warehouse you can build out of these cloud primitives to suit the specific data needs of an organisation. Think of it as a database scaled up by several orders of magnitude. Everything from storage costs to latency can be optimised as design choices. The common core in this architecture is data held in standard file formats such as parquet, delta tables, avro etc.
- DandyDev 3y agoIt’s not so bizarre if you realize that bringing ACID semantics to files, lets you use the scalability of file/blob storage like S3 combined with DB-like access. Traditional RDBMSes just don’t scale so well as S3. But S3 didn’t have ACID semantics. Now it does!
- willvarfar 3y agoSnowflake, BigQuery, Firebolt, even Trino, offer a more-classic-DB-like interface to files hosted on classic cloud storage.
- datadrivenangel 3y agoYou don't have access to the underlying data storage for Snowflake BigQuery, and Firebolt. You can import files into those, but the data will be duplicated. If you have less than a terabyte of data, using one of those data warehouses is the smart move though.
- gregw2 3y agoThis is a weird comparison to make nowadays. A more relevant question is Delta Lake vs Iceberg.
- gregw2 3y agoMrPowers, I suspect there is a case to at least attempt to make against Iceberg, but it is strange you aren't making it. Per your blog: https://mungingdata.com/devrel/virtuous-content-cycle-developer-advocates/ https://mungingdata.com/devrel/virtuous-content-cycle-develo... "This post explains how to scale developer advocacy by creating content in a way that answers current user questions and makes it easier to generate additional content in the future" As a lead for a team of developers who have used Parquet and considering Iceberg for our next-gen stuff, you aren't "answering current user questions" about whether we should consider DeltaLake, at least for me. You are marketing to a past world. Pointers from anyone on Delta vs Iceberg welcome.
- MrPowers 3y agoYea, it is fair feedback. I respect the Iceberg team & their work. I've been shying away from that post cause I don't wanna start a flamewar, but I will reflect on this and reconsider. Thank you.
- gregw2 3y agoYou are right there will be a flamewar, and others will discount some of what you say because of your bias, you will get criticism and personal remarks (mostly off base) and you will suffer tremendous heat for it. I have been there in a past life re: unix wars. But, particularly if you acknowledge opposing views in your content and don't hide counterarguments via cherry picking, you will really add value to the data community in exposing the truth, and educating people both on your team and the other team which ultimately spurs improvements where both sides have gaps and performs a greater benefit for the broader community. It takes courage and care to put a controversial rigorous viewpoint out there; you do risk your "reputation". But, particularly if you make corrections where appropriate, people will recognize you as genuine. It is not bad to have a point of view. What is bad is to hide your bias or counterarguments to deceive people. Be part of the thesis + antithesis-> synthesis Hegelian dialog that brings progress. Ultimately as you advocate for your customers (developers/data users), not "your team", you will perform a true service to the community, even if only you and a few others recognize it.
- xnx 3y agoMore comparisons (from a competitor?): "Apache Hudi vs Delta Lake vs Apache Iceberg - Data Lakehouse Feature Comparison" https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-apache-iceberg-lakehouse-feature-comparison https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-ap...
- MrPowers 3y agoLooking at this now. * Delta Lake supports merge-on-read via deletion vectors: https://delta.io/blog/2023-07-05-deletion-vectors/ https://delta.io/blog/2023-07-05-deletion-vectors/ * Why doesn't Delta Lake have efficient bulk load? Lots of the biggest datasets in the world are in Delta tables. * Delta Lake definitely supports compaction: https://delta.io/blog/2023-01-25-delta-lake-small-file-compaction-optimize/ https://delta.io/blog/2023-01-25-delta-lake-small-file-compa... * What does CLI support mean in the context of a Lakehouse storage system? You can open up a Spark shell or Python shell to interface with your Delta table. That's like saying "CSV doesn't have a CLI". I don't get it. I didn't do a detailed review of the post.
- fractaloop 3y agoIceberg (https://iceberg.apache.org https://iceberg.apache.org) is an open source alternative to Delta Lake that I cannot recommend enough. It organizes your Parquet files (or other serialization formats) in a logical structure with snapshots to allow time travel and git-like semantics for data management and Write-Audit-Publish strategies. My favorite use recently is the idempotent change data capture to ease replication in the event of failures. When your publishing job fails, you can simply replay the same diff between two snapshots and pick up where you left off.
- reactordev 3y agohttps://iceberg.apache.org https://iceberg.apache.org is the correct link.
- Eridrus 3y agoCan you share some references to the git-like semantics? I couldn't find much about merging the branched tables.
- fractaloop 3y agoAFAIK, it’s limited to fast-forward merge strategies, but you can also create or replace branches and tags, along with cherry-picking snapshots. Additional information can be found in: • the branching and tagging DDL (https://iceberg.apache.org/docs/latest/spark-ddl/#branching- https://iceberg.apache.org/docs/latest/spark-ddl/#branching-...) • Iceberg Procedures (https://iceberg.apache.org/docs/latest/spark-procedures/ https://iceberg.apache.org/docs/latest/spark-procedures/)
- BadHumans 3y agoComparing Delta Lake to Parquet is a bit nonsense isn't it? Like comparing Postgres to a zip file. After trying all of the major open table formats, Iceberg in the future in my opinion. Delta is great if you use Databricks but otherwise I don't see a compelling reason to use it over Iceberg.
- MrPowers 3y agoLots of organizations have Parquet data lakes and are considering switching to Delta Lake. Converting a Parquet table to a Delta table is an in-place, cheap computation. You can just add the Delta Lake metadata to an existing Parquet table and then take advantage of transactions and other features. I don't think it's a meaningless comparison. Iceberg is cool too.
- BadHumans 3y agoThere is no Parquet table. Parquet is a compressed file format like a zip. Parquet can be read into table formats like Hive, Delta, etc. That is why this comparison makes no sense.
- MrPowers 3y agoLots of Parquet files in the same directory are typically referred to as a "Parquet table". Yes, Parquet can be compressed with zip, but snappy is much more common because it's splittable. Parquet tables can be registered in a Hive metastore. Delta metadata can be added to a Parquet table to make it a Delta table.
- BadHumans 3y ago> Lots of Parquet files in the same directory are typically referred to as a "Parquet table". This is my point though? This is an apples to oranges comparison. A directory of Parquet files is not a table format. Comparing Delta to Hive or Iceberg is a more apt comparison. I have worked with all types of companies and I have yet to work with one that is just using a directory of Parquet files and calling it a day without using something like Hive with it.
- alexmolas 3y agoIsn't delta lake using parquet files? I don't understand the comparison. Also > Parquet tables are OK when data is in a single file but are hard to manage and unnecessarily slow when data is in many files This is not true. Having worked with Spark it's much better to have multiple "small" files than only one big file.
- MrPowers 3y agoYea, Spark works best with "right-sized" files. Let's suppose you have a data lake with 40,000 Parquet files. You need to list the files before you can read the data. This can take a few minutes. I've worked on data lakes that require file listing operations that run for hours. Key/value stores aren't good at listing files like Unix filesystems. When Spark reads the 40,000 Parquet files it needs to figure out the schema. By default, it'll just grab the schema from one of the files and just assume that all the others have the same schema. This could be wrong. You can set an option telling Spark to read the schemas of all 40,000 Parquet files and make sure they all have the same schema. That's expensive. Or you can manually specify the schema, but that can be really tedious. What if the table has 200 columns. The schema in the Parquet footer is perfect for a single file. I think storing the schema in the metadata is much better when data is spread across many Parquet files.
- adolph 3y ago> a data lake with 40,000 Parquet files. You need to list the files before you can read the data. This can take a few minutes. Sounds like this data lake could use a Parquet file listing the Parquet files. Butter
- MrPowers 3y agoYea, that's exactly what Delta Lake does. All the table metadata is stored in a Parquet file (it's initially stored in JSON files, but eventually compacted into Parquet files). These tables are sometimes so huge that the table metadata is big data also.
- jaltekruse 3y ago
- lgsilver 3y agoDatabricks has been struggling to defend Delta against the fast-moving improvements and widening adoption of Iceberg, championed by two of its major competitors, AWS and Snowflake. This article seems like a bizarre, and maybe even misleading, artifact, given that no one in the industry is comparing Parquet to Delta. They’re weighing Iceberg, which like Delta, can organize and structure groups of parquet (or other format) files…
- MrPowers 3y agoI work at Databricks, but am pretty much just an OSS nerd, mainly focusing on Delta Rust recently: https://github.com/delta-io/delta-rs https://github.com/delta-io/delta-rs I did some keyword research and wrote this post cause lots of folks are doing searches for Delta Lake vs Parquet. I'm just trying to share a fair summary of the tradeoffs with folks who are doing this search. It's a popular post and that's why I figured I would share it here.