4 ms·

Comparing Delta Lake to Parquet is a bit nonsense isn't it? Like comparing Postgres to a zip file. After trying all of the major open table formats, Iceberg in

Comparing Delta Lake to Parquet is a bit nonsense isn't it? Like comparing Postgres to a zip file. After trying all of the major open table formats, Iceberg in the future in my opinion. Delta is great if you use Databricks but otherwise I don't see a compelling reason to use it over Iceberg.

MrPowers 3y ago
Lots of organizations have Parquet data lakes and are considering switching to Delta Lake. Converting a Parquet table to a Delta table is an in-place, cheap computation. You can just add the Delta Lake metadata to an existing Parquet table and then take advantage of transactions and other features. I don't think it's a meaningless comparison. Iceberg is cool too.
- BadHumans 3y ago
  There is no Parquet table. Parquet is a compressed file format like a zip. Parquet can be read into table formats like Hive, Delta, etc. That is why this comparison makes no sense.
  - MrPowers 3y ago
    Lots of Parquet files in the same directory are typically referred to as a "Parquet table". Yes, Parquet can be compressed with zip, but snappy is much more common because it's splittable. Parquet tables can be registered in a Hive metastore. Delta metadata can be added to a Parquet table to make it a Delta table.
    - BadHumans 3y ago
      > Lots of Parquet files in the same directory are typically referred to as a "Parquet table". This is my point though? This is an apples to oranges comparison. A directory of Parquet files is not a table format. Comparing Delta to Hive or Iceberg is a more apt comparison. I have worked with all types of companies and I have yet to work with one that is just using a directory of Parquet files and calling it a day without using something like Hive with it.
      - chimerasaurus 3y ago
        I agree with the points you make above.
      - MrPowers 3y ago
        Yea, comparing Delta Lake to Iceberg is more apt, but I've been shying away from that content cause I don't wanna flamewar. Another poster is asking for this post tho, so maybe I should write it. I don't really see how Delta vs Hive comparison makes sense. A Delta table can be registered in the Hive metastore or can be unregistered. If you persist a Spark DataFrame in Delta with save it's not registered in the Hive metastore. If you persist it with saveAsTable it is registered. I've been meaning to write a blog post on this, so you're motivating me again. I've seen a bunch of enterprises that are still working with Parquet tables that aren't registered in Hive. I worked at an org like this for many years and didn't even know Hive was a thing, haha.
        BadHumans 3y ago
        > I don't really see how Delta vs Hive comparison makes sense. A Delta table can be registered in the Hive metastore or can be unregistered. You are right about Delta tables in the Hive metastore but if you are writing from the perspective of "there are companies that don't know what Hive is" then I feel the next step up is "there are companies that just stuff files in S3 and query them with Athena(which handles all the Hive stuff for you when you make tables). Explaining what Delta gives them over that I feel is something worth explaining.
bunderbunder 3y ago
I fail to see what's nonsense about comparing an extension to a format to the format it extends.
- CharlesW 3y ago
  It’s like saying, “Which is better, ISOBMFF or MPEG-4”? It’s comparing a format with an application of the format.
  - bunderbunder 3y ago
    But the article isn't just saying which is better (though I realize the article is meant as a sales pitch, so there's some of that), it's explaining what the differences are. I can't claim any sort of expertise in video codecs and file formats, but I'm guessing that one is a codec and the other is a file format that wraps the codec. If that's true, then I would say that a similar comparison between the two is also valid. Also, as someone who chooses between Parquet and Delta on a fairly regular basis, I can say from experience that there are as many situations where both are viable options than there are situations where such a comparison is invalid. So, it's hardly an apples to oranges comparison. At worst it's maybe a pomelos to oranges comparison.
    - CharlesW 3y ago
      I think the primary cognitive dissonance is that Delta Lake is a storage framework, not an open table format. It lives in a different layer of the data storage and processing stack than a file/table format like Parquet. Any time you're comparing apples to oranges, it's going to set off alarm bells with some readers. Delta Lake is a storage framework that uses Parquet files. Parquet files are a thing, but Delta Lake files are not a thing. The Delta Lake framework uses Parquet files, plus some additional stuff (transaction log, checkpoint files) that enable capabilities that Parquet files alone do not. That collection of files is called a "Delta table". If the title of this was, "Benefits of Delta Tables vs. Parquet Files Alone", and the article was revised to be more careful about not conflating Delta Lake and Parquet, I think that would benefit everyone.

MrPowers 3y ago

Lots of organizations have Parquet data lakes and are considering switching to Delta Lake. Converting a Parquet table to a Delta table is an in-place, cheap computation. You can just add the Delta Lake metadata to an existing Parquet table and then take advantage of transactions and other features. I don't think it's a meaningless comparison. Iceberg is cool too.

BadHumans 3y ago
There is no Parquet table. Parquet is a compressed file format like a zip. Parquet can be read into table formats like Hive, Delta, etc. That is why this comparison makes no sense.
- MrPowers 3y ago
  Lots of Parquet files in the same directory are typically referred to as a "Parquet table". Yes, Parquet can be compressed with zip, but snappy is much more common because it's splittable. Parquet tables can be registered in a Hive metastore. Delta metadata can be added to a Parquet table to make it a Delta table.
  - BadHumans 3y ago
    > Lots of Parquet files in the same directory are typically referred to as a "Parquet table". This is my point though? This is an apples to oranges comparison. A directory of Parquet files is not a table format. Comparing Delta to Hive or Iceberg is a more apt comparison. I have worked with all types of companies and I have yet to work with one that is just using a directory of Parquet files and calling it a day without using something like Hive with it.
    - chimerasaurus 3y ago
      I agree with the points you make above.
    - MrPowers 3y ago
      Yea, comparing Delta Lake to Iceberg is more apt, but I've been shying away from that content cause I don't wanna flamewar. Another poster is asking for this post tho, so maybe I should write it. I don't really see how Delta vs Hive comparison makes sense. A Delta table can be registered in the Hive metastore or can be unregistered. If you persist a Spark DataFrame in Delta with save it's not registered in the Hive metastore. If you persist it with saveAsTable it is registered. I've been meaning to write a blog post on this, so you're motivating me again. I've seen a bunch of enterprises that are still working with Parquet tables that aren't registered in Hive. I worked at an org like this for many years and didn't even know Hive was a thing, haha.
      - BadHumans 3y ago
        > I don't really see how Delta vs Hive comparison makes sense. A Delta table can be registered in the Hive metastore or can be unregistered. You are right about Delta tables in the Hive metastore but if you are writing from the perspective of "there are companies that don't know what Hive is" then I feel the next step up is "there are companies that just stuff files in S3 and query them with Athena(which handles all the Hive stuff for you when you make tables). Explaining what Delta gives them over that I feel is something worth explaining.

bunderbunder 3y ago

I fail to see what's nonsense about comparing an extension to a format to the format it extends.

CharlesW 3y ago
It’s like saying, “Which is better, ISOBMFF or MPEG-4”? It’s comparing a format with an application of the format.
- bunderbunder 3y ago
  But the article isn't just saying which is better (though I realize the article is meant as a sales pitch, so there's some of that), it's explaining what the differences are. I can't claim any sort of expertise in video codecs and file formats, but I'm guessing that one is a codec and the other is a file format that wraps the codec. If that's true, then I would say that a similar comparison between the two is also valid. Also, as someone who chooses between Parquet and Delta on a fairly regular basis, I can say from experience that there are as many situations where both are viable options than there are situations where such a comparison is invalid. So, it's hardly an apples to oranges comparison. At worst it's maybe a pomelos to oranges comparison.
  - CharlesW 3y ago
    I think the primary cognitive dissonance is that Delta Lake is a storage framework, not an open table format. It lives in a different layer of the data storage and processing stack than a file/table format like Parquet. Any time you're comparing apples to oranges, it's going to set off alarm bells with some readers. Delta Lake is a storage framework that uses Parquet files. Parquet files are a thing, but Delta Lake files are not a thing. The Delta Lake framework uses Parquet files, plus some additional stuff (transaction log, checkpoint files) that enable capabilities that Parquet files alone do not. That collection of files is called a "Delta table". If the title of this was, "Benefits of Delta Tables vs. Parquet Files Alone", and the article was revised to be more careful about not conflating Delta Lake and Parquet, I think that would benefit everyone.