10 ms·

The two versions of Parquet

206 points by tanelpoder 1y ago

blntechie 1y ago
What does DuckDB mean by query engines here? Something like Trino?
- dkdcio 1y ago
  some sources: - https://voltrondata.com/codex/a-new-frontier https://voltrondata.com/codex/a-new-frontier (links out to others) - https://wesmckinney.com/blog/looking-back-15-years/ https://wesmckinney.com/blog/looking-back-15-years/ in short you can think of a DB as at least 3 decoupled subsystems: UI, compute (query engine), storage. DuckDB has a query engine and storage format, and several UIs (SQL, Python, etc.). Trino is only a query engine (and UIs, everything has UIs). Polars has a query engine. DataFusion is a query engine (and other things). Spark is a query engine. pandas has a query engine typically query engines are tightly coupled with the overall “product”, but increasingly compute, data (and even more recently via DuckLake metadata), and UI are decoupled allowing you to mix and match parts for a “decomposed database” architecture quick disclaimer: I worked at Voltron Data but it’s a dead company walking, not trying to advertise for them by any means but the doc I linked is very well written with good information IMO
crmd 1y ago
I am saying this as a lifelong supporter and user of open source software: issues like this are why governments and enterprises still run on Oracle and SQL Server. The author was able to rollback his changes, but in some industries an unplanned enterprise-wide data unavailability event means the end of your career at that firm, if you don’t have a CYA email from the vendor confirming you were good to go. That CYA email, and the throat to choke, is why Oracle does 7 and 8 figure licensing deals with enterprises selling inferior software solutions versus open source options. It seems that Linux, through Linus’ leadership, has been able to solve this risk issue and fully displace commercial UNIX operating systems. I hope many other projects up and down the stack can have the same success.
- duncanfwalker 1y ago
  At the start of your comment I thought the 'issues like this' were going to be the 4 year discussions about what is and isn't core.
  - crmd 1y ago
    So did I :-) but I think the concepts are related: Linus’ ability to shift into autocratic leadership mode when necessary seems to prevent issues like the 4 year indecisiveness on v2/core from compromising product quality to the point where Linux is trusted in a way that rivals commercial software.
    - duncanfwalker 1y ago
      +1 you're paying for the governance as much as you're paying for the code.
      - crmd 1y ago
        Well said, thank you
- moelf 1y ago
  and why CERN rocking their own file format, again in, 2025, https://cds.cern.ch/record/2923186 https://cds.cern.ch/record/2923186
  - 3eb7988a1663 1y ago
    To be fair, CERNs needs do seem fairly niche. Petabyte numeric datasets with all sorts of access patterns from researchers. All of which they want to maintain compatible software forever.
    - moelf 1y ago
      yeah except this new RNTuple thing is really really similar to Apache Arrow
viccis 1y ago
Yeah I had to wait years to really use Parquet effectively in Python code back in the 2010s because there were two main ones (Pyarrow and Fastparquet), and they were neither compatible with either other nor compatible with Spark. Parquet support is much like Javascript support in browsers. You only get to use the more advanced features when they are supported compatibly on every platform you expect them to be used.
lowbloodsugar 1y ago
When working with your own datasets, v2 is a must. If you are willing to make trade offs you can get insane compression and speed.
- ted_dunning 1y ago
  Why doesn't this show in the examples in the article? Do you have examples?
  - lowbloodsugar 1y ago
    Couple of examples can think of off the top of my head from recording logs for analysis. 1. It might be better to buffer up logs, then sort on a different column than time. You may benefit from the delta encoding or prefix encoding. 2. If you have tracking info, which is usually a random like a UUID or something, then ditch it. Not debugging with this dataset so don’t waste the space on a crazy high noise column. Shit like that.
sighansen 1y ago
As long as iceberg and delta lake won't support v2, adoption will be really hard. I'm working aot with parquet and wasn't even aware that there is a version 2.0.
- lolive 1y ago
  Why wouldn't they adopt the v2.0?
  - mr_toad 1y ago
    Version 1 took about ten years before it became de rigueur. Version 2 is hot off the press.
    - lolive 1y ago
      From my memories, when Unicode arrived [i.e ages ago], I bet 10$ it would never succeed . Now that it is reasonably supported everywhere [and I lost my 10$], I am more confident that sometimes good ideas eventually win. #callMeOptimist
sbassi 1y ago
Shameless plug: made a parquet conversion utility: pip install parquetconv It is a command line wrapper to generate a Pandas SF and save it as CSV (or the other way around)
1a527dd5 1y ago
https://www.jeronimo.dev/the-two-versions-of-parquet/#performance-of-version-2 https://www.jeronimo.dev/the-two-versions-of-parquet/#perfor... First paragraph under that heading as a markdown error which I hadn’t considered in [my previous post on compression algorithms]](/compression-algorithms-parquet/).
adrian17 1y ago
I was quite confused when I learned that the spec technically supports metadata about whether the data is already pre-sorted by some column(s); in my eyes seemed like it would allow some non-brainer optimizations. And yet, last I checked, it looked like pretty much nothing actually uses it, and some libraries don't even read this field at all.
mgaunard 1y ago
Arrow defaults to v2.6, and I've seen a few places downgrade to 2.4 for compatibility. Never seen any v1 in the wild.
- ayhanfuat 1y ago
  They are mostly talking about new encodings and those are controlled by data_page_version which still defaults to v1. The one you are talking about is about schema and types. I guess that is easier to handle.
nly 1y ago
This sort of problem is common with file formats that reach popularity <= V1 and then don't iterate quickly. "Simple Binary Encoding" v2 has been stuck at release candidate stage for over 5 years and has flaws that mean it'll probably never be worth adopting.
arecurrence 1y ago
Sounds similar to HDMI allowing varying levels of specification completeness to all be called the same thing.
- mr_toad 1y ago
  The Hitchhiker's Guide to the Galaxy defines the marketing division of the Sirius Cybernetics Corporation as "a bunch of mindless jerks who'll be the first against the wall when the revolution comes,"
quotemstr 1y ago
> Although this post might seem like a critique of Parquet, that is not my intention. I am simply documenting what I have learned and explaining the challenges maintainers of an open format face when evolving it. All the benefits and utilities that a format like Parquet has far outweigh these inconveniences. Yes, it is a critique (or at least its user community). It's a critique that's 100% justified too. Have we all been so conditioned by corporate training that we've lost the ability to say "hey, this sucks" when it _does_ in fact suck? We all lose when people communicate unclearly. Here, the people holding back evolution of the format do need to be critiqued, and named, and shamed, and the author shouldn't have been so shy about doing it.
willtemperley 1y ago
The reference implementation for Parquet is a gigantic Java library. I'm unconvinced this is a good idea. Take the RLE encoding which switches between run-length encoding and bit-packing. The way bit-packing has been implemented is to generate 74,000 lines of Java to read/write every combination of bitwidth, endianness and value-length. I just can't believe this is optimal except in maybe very specific CPU-only cases (e.g. Parquet-Java running on a giant cluster somewhere). If it were just bit-packing I could easily offload a whole data page to a GPU and not care about having per-bitwidth optimised implementations, but having to switch encodings at random intervals just makes this a headache. It would be really nice if actual design documents exist that specify why this is a good idea based on real-world data patterns.
- willtemperley 1y ago
  Addendum: if something is actually decoded by RunLengthBitPackingHybridDecoder but you call the encoding RLE this is probably because it was a bad idea in the first place. Plus it makes it really hard to search for.
- ignoreusernames 1y ago
  > The reference implementation for Parquet is a gigantic Java library. I'm unconvinced this is a good idea. I haven't though much about it, but I believe the ideal reference implementation would be a highly optimized "service like" process that you run alongside your engine using arrow to share zero copy buffers between the engine and the parquet service. Parquet predates arrow by quite a few years and java was (unfortunately) the standard for big data stuff back then, so they simply stuck with it. > The way bit-packing has been implemented is to generate 74,000 lines of Java to read/write every combination of bitwidth, endianness and value-length I think they did this to avoid the dynamic dispatch nature of java. If using C++ or Rust something very similar would happen, but at the compiler level which is a much saner way of doing this kind of thing.
  - willtemperley 1y ago
    Actually looking at the DuckDB source I think they re-use a single uint64 and push bits onto this a byte at a time, until bitwidth is reached, then right-shift bitwidth bits back off when a single value has been created. Very neat and presumably quick. I've just had so many issues with total lack of clarity with this format. They tell you a total_compressed_size for a page then it turns out the _uncompressed_ page header is included in this - but the documentation barely give any clues to the layout [1]. The reality: Each column chunk includes a list of pages written back-to-back, with an optional dictionary page first. Each of these, including the dictionary are prepended with an uncompressed PageHeader in Thrift format. It wasn't too hard to write a paragraph about it. It was quite hard looking for magic compression bytes in hex dumps. Maybe there should be a "minimum workable reference implementation" or something that is slow but easy to understand. [1] https://parquet.apache.org/docs/file-format/data-pages/columnchunks/ https://parquet.apache.org/docs/file-format/data-pages/colum...
Uehreka 1y ago
Only one can win. Only one can be allowed to live. To decide which one, I hereby convene… (slams gavel) Parquet Court.
atbpaca 1y ago
Similarly, Apache Spark and Scala versions. Spark ran on Scala 2.12 for a long time to eventually support 2.13. To this day, no plans to support Scala 3.x. Databricks started supporting 2.13 only in May this year...
sonium 1y ago
TLDR: There are two versions of the Parquet file format, but adoption of Version 2 is slow due to limited compatibility in major engines and tools. While Version 2 offers improvements (smaller file sizes, faster write/read times), these gains are modest, and ecosystem support remains fragmented. If full control over the data pipeline is possible, using Version 2 can be worthwhile; otherwise, compatibility concerns with third-party integrations may outweigh the benefits. Parquet remains dominant, and its utility far surpasses these challenges
paparicio 1y ago
I always follow all of your posts. Parquet is amazinggggggg

206 points by tanelpoder 1y ago

blntechie 1y ago

What does DuckDB mean by query engines here? Something like Trino?

dkdcio 1y ago
some sources: - https://voltrondata.com/codex/a-new-frontier https://voltrondata.com/codex/a-new-frontier (links out to others) - https://wesmckinney.com/blog/looking-back-15-years/ https://wesmckinney.com/blog/looking-back-15-years/ in short you can think of a DB as at least 3 decoupled subsystems: UI, compute (query engine), storage. DuckDB has a query engine and storage format, and several UIs (SQL, Python, etc.). Trino is only a query engine (and UIs, everything has UIs). Polars has a query engine. DataFusion is a query engine (and other things). Spark is a query engine. pandas has a query engine typically query engines are tightly coupled with the overall “product”, but increasingly compute, data (and even more recently via DuckLake metadata), and UI are decoupled allowing you to mix and match parts for a “decomposed database” architecture quick disclaimer: I worked at Voltron Data but it’s a dead company walking, not trying to advertise for them by any means but the doc I linked is very well written with good information IMO

crmd 1y ago

I am saying this as a lifelong supporter and user of open source software: issues like this are why governments and enterprises still run on Oracle and SQL Server. The author was able to rollback his changes, but in some industries an unplanned enterprise-wide data unavailability event means the end of your career at that firm, if you don’t have a CYA email from the vendor confirming you were good to go. That CYA email, and the throat to choke, is why Oracle does 7 and 8 figure licensing deals with enterprises selling inferior software solutions versus open source options. It seems that Linux, through Linus’ leadership, has been able to solve this risk issue and fully displace commercial UNIX operating systems. I hope many other projects up and down the stack can have the same success.

duncanfwalker 1y ago
At the start of your comment I thought the 'issues like this' were going to be the 4 year discussions about what is and isn't core.
- crmd 1y ago
  So did I :-) but I think the concepts are related: Linus’ ability to shift into autocratic leadership mode when necessary seems to prevent issues like the 4 year indecisiveness on v2/core from compromising product quality to the point where Linux is trusted in a way that rivals commercial software.
  - duncanfwalker 1y ago
    +1 you're paying for the governance as much as you're paying for the code.
    - crmd 1y ago
      Well said, thank you
moelf 1y ago
and why CERN rocking their own file format, again in, 2025, https://cds.cern.ch/record/2923186 https://cds.cern.ch/record/2923186
- 3eb7988a1663 1y ago
  To be fair, CERNs needs do seem fairly niche. Petabyte numeric datasets with all sorts of access patterns from researchers. All of which they want to maintain compatible software forever.
  - moelf 1y ago
    yeah except this new RNTuple thing is really really similar to Apache Arrow

viccis 1y ago

Yeah I had to wait years to really use Parquet effectively in Python code back in the 2010s because there were two main ones (Pyarrow and Fastparquet), and they were neither compatible with either other nor compatible with Spark. Parquet support is much like Javascript support in browsers. You only get to use the more advanced features when they are supported compatibly on every platform you expect them to be used.

lowbloodsugar 1y ago

When working with your own datasets, v2 is a must. If you are willing to make trade offs you can get insane compression and speed.

ted_dunning 1y ago
Why doesn't this show in the examples in the article? Do you have examples?
- lowbloodsugar 1y ago
  Couple of examples can think of off the top of my head from recording logs for analysis. 1. It might be better to buffer up logs, then sort on a different column than time. You may benefit from the delta encoding or prefix encoding. 2. If you have tracking info, which is usually a random like a UUID or something, then ditch it. Not debugging with this dataset so don’t waste the space on a crazy high noise column. Shit like that.

sighansen 1y ago

As long as iceberg and delta lake won't support v2, adoption will be really hard. I'm working aot with parquet and wasn't even aware that there is a version 2.0.

lolive 1y ago
Why wouldn't they adopt the v2.0?
- mr_toad 1y ago
  Version 1 took about ten years before it became de rigueur. Version 2 is hot off the press.
  - lolive 1y ago
    From my memories, when Unicode arrived [i.e ages ago], I bet 10$ it would never succeed . Now that it is reasonably supported everywhere [and I lost my 10$], I am more confident that sometimes good ideas eventually win. #callMeOptimist

sbassi 1y ago

Shameless plug: made a parquet conversion utility: pip install parquetconv It is a command line wrapper to generate a Pandas SF and save it as CSV (or the other way around)

1a527dd5 1y ago

https://www.jeronimo.dev/the-two-versions-of-parquet/#performance-of-version-2 https://www.jeronimo.dev/the-two-versions-of-parquet/#perfor... First paragraph under that heading as a markdown error which I hadn’t considered in [my previous post on compression algorithms]](/compression-algorithms-parquet/).

adrian17 1y ago

I was quite confused when I learned that the spec technically supports metadata about whether the data is already pre-sorted by some column(s); in my eyes seemed like it would allow some non-brainer optimizations. And yet, last I checked, it looked like pretty much nothing actually uses it, and some libraries don't even read this field at all.

mgaunard 1y ago

Arrow defaults to v2.6, and I've seen a few places downgrade to 2.4 for compatibility. Never seen any v1 in the wild.

ayhanfuat 1y ago
They are mostly talking about new encodings and those are controlled by data_page_version which still defaults to v1. The one you are talking about is about schema and types. I guess that is easier to handle.

nly 1y ago

This sort of problem is common with file formats that reach popularity <= V1 and then don't iterate quickly. "Simple Binary Encoding" v2 has been stuck at release candidate stage for over 5 years and has flaws that mean it'll probably never be worth adopting.

arecurrence 1y ago

Sounds similar to HDMI allowing varying levels of specification completeness to all be called the same thing.

mr_toad 1y ago
The Hitchhiker's Guide to the Galaxy defines the marketing division of the Sirius Cybernetics Corporation as "a bunch of mindless jerks who'll be the first against the wall when the revolution comes,"

quotemstr 1y ago

> Although this post might seem like a critique of Parquet, that is not my intention. I am simply documenting what I have learned and explaining the challenges maintainers of an open format face when evolving it. All the benefits and utilities that a format like Parquet has far outweigh these inconveniences. Yes, it is a critique (or at least its user community). It's a critique that's 100% justified too. Have we all been so conditioned by corporate training that we've lost the ability to say "hey, this sucks" when it _does_ in fact suck? We all lose when people communicate unclearly. Here, the people holding back evolution of the format do need to be critiqued, and named, and shamed, and the author shouldn't have been so shy about doing it.

willtemperley 1y ago

The reference implementation for Parquet is a gigantic Java library. I'm unconvinced this is a good idea. Take the RLE encoding which switches between run-length encoding and bit-packing. The way bit-packing has been implemented is to generate 74,000 lines of Java to read/write every combination of bitwidth, endianness and value-length. I just can't believe this is optimal except in maybe very specific CPU-only cases (e.g. Parquet-Java running on a giant cluster somewhere). If it were just bit-packing I could easily offload a whole data page to a GPU and not care about having per-bitwidth optimised implementations, but having to switch encodings at random intervals just makes this a headache. It would be really nice if actual design documents exist that specify why this is a good idea based on real-world data patterns.

willtemperley 1y ago
Addendum: if something is actually decoded by RunLengthBitPackingHybridDecoder but you call the encoding RLE this is probably because it was a bad idea in the first place. Plus it makes it really hard to search for.
ignoreusernames 1y ago
> The reference implementation for Parquet is a gigantic Java library. I'm unconvinced this is a good idea. I haven't though much about it, but I believe the ideal reference implementation would be a highly optimized "service like" process that you run alongside your engine using arrow to share zero copy buffers between the engine and the parquet service. Parquet predates arrow by quite a few years and java was (unfortunately) the standard for big data stuff back then, so they simply stuck with it. > The way bit-packing has been implemented is to generate 74,000 lines of Java to read/write every combination of bitwidth, endianness and value-length I think they did this to avoid the dynamic dispatch nature of java. If using C++ or Rust something very similar would happen, but at the compiler level which is a much saner way of doing this kind of thing.
- willtemperley 1y ago
  Actually looking at the DuckDB source I think they re-use a single uint64 and push bits onto this a byte at a time, until bitwidth is reached, then right-shift bitwidth bits back off when a single value has been created. Very neat and presumably quick. I've just had so many issues with total lack of clarity with this format. They tell you a total_compressed_size for a page then it turns out the _uncompressed_ page header is included in this - but the documentation barely give any clues to the layout [1]. The reality: Each column chunk includes a list of pages written back-to-back, with an optional dictionary page first. Each of these, including the dictionary are prepended with an uncompressed PageHeader in Thrift format. It wasn't too hard to write a paragraph about it. It was quite hard looking for magic compression bytes in hex dumps. Maybe there should be a "minimum workable reference implementation" or something that is slow but easy to understand. [1] https://parquet.apache.org/docs/file-format/data-pages/columnchunks/ https://parquet.apache.org/docs/file-format/data-pages/colum...

Uehreka 1y ago

Only one can win. Only one can be allowed to live. To decide which one, I hereby convene… (slams gavel) Parquet Court.

atbpaca 1y ago

Similarly, Apache Spark and Scala versions. Spark ran on Scala 2.12 for a long time to eventually support 2.13. To this day, no plans to support Scala 3.x. Databricks started supporting 2.13 only in May this year...

sonium 1y ago

TLDR: There are two versions of the Parquet file format, but adoption of Version 2 is slow due to limited compatibility in major engines and tools. While Version 2 offers improvements (smaller file sizes, faster write/read times), these gains are modest, and ecosystem support remains fragmented. If full control over the data pipeline is possible, using Version 2 can be worthwhile; otherwise, compatibility concerns with third-party integrations may outweigh the benefits. Parquet remains dominant, and its utility far surpasses these challenges

paparicio 1y ago

I always follow all of your posts. Parquet is amazinggggggg