7 ms·

The Birth of Parquet

145 points by whinvik 2y ago

jjgreen 2y ago
Why is it not in Debian? Is there some deep and dark secret? https://search.debian.org/cgi-bin/omega?DB=en&P=parquet https://search.debian.org/cgi-bin/omega?DB=en&P=parquet
- whinvik 2y ago
  Parquet is a file format. Should a file format be in Debian?
  - jjgreen 2y ago
    I mean library support for reading and writing it: > apt-cache search hdf5 | wc -l 134 > apt-cache search netcdf | wc -l 70 > apt-cache search parquet | wc -l 0
    - jjtheblunt 2y ago
      look for polars, https://pola.rs https://pola.rs
      - jjgreen 2y ago
        https://packages.debian.org/search?searchon=names&keywords=polars https://packages.debian.org/search?searchon=names&keywords=p...
        petre 2y ago
        Use pip, Luke! https://docs.pola.rs/user-guide/getting-started https://docs.pola.rs/user-guide/getting-started
        jjgreen 2y ago
        My question is "why isn't it in Debian?", I ask that since Debian has rather high standards and the absence from Debian suggests some quality issue in available libraries for the format or the format itself. Are there dark secrets?
        jjtheblunt 2y ago
        Could the dark secret you seek be "debian isn't for bleeding edge packages"? it's very modern and perhaps hasn't been around long enough to have debian maintainers feel it's vetted. for instance, documentation for Python bindings is more advanced than for Rust bindings, but the package itself uses Rust at the low level.
        jjgreen 2y ago
        Parquet is what, 12 years old? Hardly cutting edge. What you say my well be true for polars (I'm not familiar with it), if/when it (or something else) does get packaged I'll give parquet another look ...
        jjtheblunt 2y ago
        Yes, i wasn't clear: it's the polars library that's actively changing, so that might be the issue, or just the vast set of optional components configurable on installation, which isn't the normal package manager experience. FWIW i think i share your general aversion to _not_ using packages, just for the tidiness of installs and removals, though i'm on fedora and macos.
        petre 2y ago
        Pandas is probably in Debian and it can read parquet files. Polars is fairly new and under active development. It's a python library, I install those in $HOME/.local, as opposed to system wide. One can also install it in a venv. With pip you can also uninstall packages and keep things fairly tidy.
        jjtheblunt 2y ago
        Why did you ignore the link i gave you above? If you follow that link, you'll see polars and parquet are a large highly configurable collection of tools for format manipulations across many HPC formats. Debian maintainers possibly don't want to bundle the entirety, as it would be vast. Might this help you, though? https://cloudsmith.io/~opencpn/repos/polar-prod/packages/detail/raw/polar_pi-1.2.6.0-debian-x86_64-10-buster-tarball/v1.2.6.0/ https://cloudsmith.io/~opencpn/repos/polar-prod/packages/det...
    - whinvik 2y ago
      Are there Arrow libraries. Feel like a lot of applications that read Parquet actually outsource raw reads to Arrow.
    - marginalia_nu 2y ago
      Duckdb is probably what you want, though I don't think it's in debian either. It's in Arch though.
- mistrial9 2y ago
  pandas is a python-centric, tabular data handler that works well in clouds (and desktop Debian). Pandas can read parquet data today, among other libs mentioned. The binary dot-so driver style is single-host centric and not the emphasis of these cloudy projects (and their cloudy funders) https://pandas.pydata.org/docs/reference/api/pandas.read_parquet.html https://pandas.pydata.org/docs/reference/api/pandas.read_par... https://packages.debian.org/buster/python3-pandas https://packages.debian.org/buster/python3-pandas Perhaps more alarm is called for when this python+pandas and parquet does not work on Debian, but that is not the case today. ps- data access in clouds often uses the S3:// endpoint . Contrast to a POSIX endpoint using _fread()_ or similar.. many parquet-aware clients prefer the cloudy, un-POSIX method to access data and that is another reason it is not a simple package in Debian today.
  - jjgreen 2y ago
    As I understand it, pandas can read parquet if the pyarrow or fastparquet packages are available, but that's not the case and attempts to fix that have been underway for several years. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021
    - mistrial9 2y ago
      hm https://github.com/pandas-dev/pandas/blob/main/pandas/io/parquet.py https://github.com/pandas-dev/pandas/blob/main/pandas/io/par... says "fastparquet" engine must be available if no pyarrow https://pypi.org/project/fastparquet/ https://pypi.org/project/fastparquet/
      - jjgreen 2y ago
        See https://news.ycombinator.com/item?id=40306445 https://news.ycombinator.com/item?id=40306445
  - datadrivenangel 2y ago
    Pandas often has significant memory overhead, so it's not uncommon to need ~3-5x the amount of memory as your file size. Polars and DuckDB are much better about memory management.
- pabs3 2y ago
  No-one with the combination of motivation, time and skills needed get it into Debian. Someone wanted to get a Python implementation in, but it looks like they never found the time. https://bugs.debian.org/838338 https://bugs.debian.org/838338 These days Debian packaging has become a bit irrelevant, since you can just shove upstream releases into a container and go for it.
  - mkesper 2y ago
    If you want to depend on curl|bash setups, sure. If you want to get security updates, good packaging is still required.
calderwoodra 2y ago
I've been struggeling with a tough parquet problem for a few months now. I have a 15gb parquet file in a s3 bucket and I need to "unzip" and extract every row from the file to write into my database. The contents of the file are emails and I need to integrate them into our search function. Is this possible to do without an unreasonable amount of RAM? Are there any affordable services that can help here? Feel free to contact me (email in bio), happy to pay for a consult at the minimum.
- martinky24 2y ago
  Wouldn't partial reads fix the RAM problem? e.g. something like this: https://stackoverflow.com/a/69888274 https://stackoverflow.com/a/69888274 It might not be fast, but a quick 1-off solution that you let run for a while would probably do that job. There shouldn't be a need to load the whole file into memory.
- memset 2y ago
  I can help! Emailing you now :) Our company (scratchdata.com, open source) is literally built to solve the problem of schlepping large amounts of data between sources and destinations, so I have worked on this problem a lot personally and happy to nerd out about what works.
  - fock 2y ago
    I - by my HPC-background - am wondering quite a bit what happened that 15GB-files are considered large data? Not being a crazy parquet-user, but: - does this decompress to giant sizes? - can't you split the file easily, because it includes row-based segments? - why does it take months to solve this for one file?
    - semi-extrinsic 2y ago
      As a fellow HPC user, I tried a couple of years ago to do a tricky data format conversion using these newfangled tools. I was essentially just taking a huge (multi-terabyte) 3D dataset, transposing it and changing the endianness. The solutions I was able to put together using Dask and Spark and such were all insanely slow, they just got killed by Slurm without getting anywhere. In the end I went back to good ole' shell scripting with xxd to handle most of the heavy lifting. Finished in under an hour. The appeal of these newfangled tools is that you can work with data sizes that are infeasible to people who only know Excel, yet you don't need to understand a single thing about how your data is actually stored. If you can be bothered to read the file format specification, open up some files in a hex editor to understand the layout, and write low-level code to parse the data - then you can achieve several orders of magnitude higher performance.
fizx 2y ago
Fun story time: I was at twitter for a few years and tended to write quick hacks that people wanted to replace with better engineering. We never had scala thrift bindings, and the Java ones were awkward from Scala, so I wrote a thrift plugin in JRuby that used the Ruby thrift parser and ERb web templates to output some Scala code. Integrated with our build pipeline, worked great for the company. I also wrote one era of twitter's service deploy system on a hacked up Capistrano. These projects took a few days because they were dirty hacks, but I still got a below perf review for getting easily distracted, because I didn't yet know how to sell those company-wide projects. Anyhow, about a month before that team kicked off Parquet, I showed them a columnar format I made for a hackweek based on Lucene's codec packages, and was using to power a mixpanel-alike analytics system. I'm not sure whether they were inspired or terrified that my hack would reach production, but I like to think I had a small hand in getting Parquet kickstarted.
- hi-v-rocknroll 2y ago
  Hahaha. Don't ask how the sausage gets made, so long as the sausage machine is readily maintainable by more than one person or has documentation. Partially explains how murder came to be https://github.com/lg/murder https://github.com/lg/murder > below perf review That's some cheap bullshit. Fuck marketing-oriented corporate engineering.
pradeepchhetri 2y ago
Reading through this blog, to me it seems Parquet is lot like ClickHouse native data format. Best part of ClickHouse native data format is I can use the same ClickHouse queries and can run in local or remote server/cluster and let ClickHouse to decide the available resources in the most performant way. ClickHouse has a native and the fastest integration with Parquet so i can: - Query local/s3 parquet data from command line using clickhouse-local. - Query large amount of local/s3 data programmatically by offloading it to clickhouse server/cluster which can do processing in distributed fashion.
- pradeepchhetri 2y ago
  If you are interested in reading internals of using Parquet with ClickHouse, do read following articles: - https://clickhouse.com/blog/apache-parquet-clickhouse-local-querying-writing https://clickhouse.com/blog/apache-parquet-clickhouse-local-... - https://clickhouse.com/blog/apache-parquet-clickhouse-local-querying-writing-internals-row-groups https://clickhouse.com/blog/apache-parquet-clickhouse-local-...
colloydi 2y ago
Tangential to the topic but regarding the supposed Snowball Effect there is in real life no such thing. I have pushed large 'snowballs' down slopes --in reality they are snow cylinders as shown in the photo-- and they invariably do not get far. The reason being that when one side of the cylinder randomly thickens slightly with respect to the other side this causes the whole thing to turn in the opposite direction. For example, if the RHS of your cylinder has a slightly larger radius than the LHS the cylinder will commence turning to the left. The upshot is the thick side picks up more snow than the thin side and the disparity in radii increases more rapidly still. The cylinder becomes a truncated cone which turns sideways and halts!
- whinvik 2y ago
  Sounds to me like there's a long blog post waiting to be written.
- lsowen 2y ago
  It might be rare, but it _can_ happen: https://www.youtube.com/watch?v=ZdIclqha014 https://www.youtube.com/watch?v=ZdIclqha014
- dendrite9 2y ago
  It is highly dependent on the snow conditions and the recent weather. Sometimes even just the a couple hours are enough to change the conditions to have a good chance of rollerballs. The climate also has an impact, in my experience more coastal areas have more periods when they form. And in some cases the rollerballs get too tall for the bonding strength of the snow, so they break into parts that can restart the cycle if the slope is steep enough.
anentropic 2y ago
Following along the subsequent blog posts in this series progress to Arrow and then to... OpenLineage I'm curious if anyone has experience with OpenLineage/Marquez or similar they'd like to share
Zababa 2y ago
It's very interesting to see how a new "enterprise open source" project is born. The part where right at the start the author knows that they should have more than one company on board, and how the other companies each contribute their part.
heroprotagonist 2y ago
> most basic definition of open source is the code is available for you to read What crap. That's 'source-available', NOT open-source. But at least co-option of terminology is an indicator of success.

145 points by whinvik 2y ago

Why is it not in Debian? Is there some deep and dark secret? https://search.debian.org/cgi-bin/omega?DB=en&P=parquet https://search.debian.org/cgi-bin/omega?DB=en&P=parquet

whinvik 2y ago
Parquet is a file format. Should a file format be in Debian?
- jjgreen 2y ago
  I mean library support for reading and writing it: > apt-cache search hdf5 | wc -l 134 > apt-cache search netcdf | wc -l 70 > apt-cache search parquet | wc -l 0
  - jjtheblunt 2y ago
    look for polars, https://pola.rs https://pola.rs
    - jjgreen 2y ago
      https://packages.debian.org/search?searchon=names&keywords=polars https://packages.debian.org/search?searchon=names&keywords=p...
      - petre 2y ago
        Use pip, Luke! https://docs.pola.rs/user-guide/getting-started https://docs.pola.rs/user-guide/getting-started
        jjgreen 2y ago
        My question is "why isn't it in Debian?", I ask that since Debian has rather high standards and the absence from Debian suggests some quality issue in available libraries for the format or the format itself. Are there dark secrets?
        jjtheblunt 2y ago
        Could the dark secret you seek be "debian isn't for bleeding edge packages"? it's very modern and perhaps hasn't been around long enough to have debian maintainers feel it's vetted. for instance, documentation for Python bindings is more advanced than for Rust bindings, but the package itself uses Rust at the low level.
        jjgreen 2y ago
        Parquet is what, 12 years old? Hardly cutting edge. What you say my well be true for polars (I'm not familiar with it), if/when it (or something else) does get packaged I'll give parquet another look ...
        jjtheblunt 2y ago
        Yes, i wasn't clear: it's the polars library that's actively changing, so that might be the issue, or just the vast set of optional components configurable on installation, which isn't the normal package manager experience. FWIW i think i share your general aversion to _not_ using packages, just for the tidiness of installs and removals, though i'm on fedora and macos.
        petre 2y ago
        Pandas is probably in Debian and it can read parquet files. Polars is fairly new and under active development. It's a python library, I install those in $HOME/.local, as opposed to system wide. One can also install it in a venv. With pip you can also uninstall packages and keep things fairly tidy.
      - jjtheblunt 2y ago
        Why did you ignore the link i gave you above? If you follow that link, you'll see polars and parquet are a large highly configurable collection of tools for format manipulations across many HPC formats. Debian maintainers possibly don't want to bundle the entirety, as it would be vast. Might this help you, though? https://cloudsmith.io/~opencpn/repos/polar-prod/packages/detail/raw/polar_pi-1.2.6.0-debian-x86_64-10-buster-tarball/v1.2.6.0/ https://cloudsmith.io/~opencpn/repos/polar-prod/packages/det...
  - whinvik 2y ago
    Are there Arrow libraries. Feel like a lot of applications that read Parquet actually outsource raw reads to Arrow.
  - marginalia_nu 2y ago
    Duckdb is probably what you want, though I don't think it's in debian either. It's in Arch though.
mistrial9 2y ago
pandas is a python-centric, tabular data handler that works well in clouds (and desktop Debian). Pandas can read parquet data today, among other libs mentioned. The binary dot-so driver style is single-host centric and not the emphasis of these cloudy projects (and their cloudy funders) https://pandas.pydata.org/docs/reference/api/pandas.read_parquet.html https://pandas.pydata.org/docs/reference/api/pandas.read_par... https://packages.debian.org/buster/python3-pandas https://packages.debian.org/buster/python3-pandas Perhaps more alarm is called for when this python+pandas and parquet does not work on Debian, but that is not the case today. ps- data access in clouds often uses the S3:// endpoint . Contrast to a POSIX endpoint using _fread()_ or similar.. many parquet-aware clients prefer the cloudy, un-POSIX method to access data and that is another reason it is not a simple package in Debian today.
- jjgreen 2y ago
  As I understand it, pandas can read parquet if the pyarrow or fastparquet packages are available, but that's not the case and attempts to fix that have been underway for several years. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021
  - mistrial9 2y ago
    hm https://github.com/pandas-dev/pandas/blob/main/pandas/io/parquet.py https://github.com/pandas-dev/pandas/blob/main/pandas/io/par... says "fastparquet" engine must be available if no pyarrow https://pypi.org/project/fastparquet/ https://pypi.org/project/fastparquet/
    - jjgreen 2y ago
      See https://news.ycombinator.com/item?id=40306445 https://news.ycombinator.com/item?id=40306445
- datadrivenangel 2y ago
  Pandas often has significant memory overhead, so it's not uncommon to need ~3-5x the amount of memory as your file size. Polars and DuckDB are much better about memory management.
pabs3 2y ago
No-one with the combination of motivation, time and skills needed get it into Debian. Someone wanted to get a Python implementation in, but it looks like they never found the time. https://bugs.debian.org/838338 https://bugs.debian.org/838338 These days Debian packaging has become a bit irrelevant, since you can just shove upstream releases into a container and go for it.
- mkesper 2y ago
  If you want to depend on curl|bash setups, sure. If you want to get security updates, good packaging is still required.

calderwoodra 2y ago

I've been struggeling with a tough parquet problem for a few months now. I have a 15gb parquet file in a s3 bucket and I need to "unzip" and extract every row from the file to write into my database. The contents of the file are emails and I need to integrate them into our search function. Is this possible to do without an unreasonable amount of RAM? Are there any affordable services that can help here? Feel free to contact me (email in bio), happy to pay for a consult at the minimum.

martinky24 2y ago
Wouldn't partial reads fix the RAM problem? e.g. something like this: https://stackoverflow.com/a/69888274 https://stackoverflow.com/a/69888274 It might not be fast, but a quick 1-off solution that you let run for a while would probably do that job. There shouldn't be a need to load the whole file into memory.
memset 2y ago
I can help! Emailing you now :) Our company (scratchdata.com, open source) is literally built to solve the problem of schlepping large amounts of data between sources and destinations, so I have worked on this problem a lot personally and happy to nerd out about what works.
- fock 2y ago
  I - by my HPC-background - am wondering quite a bit what happened that 15GB-files are considered large data? Not being a crazy parquet-user, but: - does this decompress to giant sizes? - can't you split the file easily, because it includes row-based segments? - why does it take months to solve this for one file?
  - semi-extrinsic 2y ago
    As a fellow HPC user, I tried a couple of years ago to do a tricky data format conversion using these newfangled tools. I was essentially just taking a huge (multi-terabyte) 3D dataset, transposing it and changing the endianness. The solutions I was able to put together using Dask and Spark and such were all insanely slow, they just got killed by Slurm without getting anywhere. In the end I went back to good ole' shell scripting with xxd to handle most of the heavy lifting. Finished in under an hour. The appeal of these newfangled tools is that you can work with data sizes that are infeasible to people who only know Excel, yet you don't need to understand a single thing about how your data is actually stored. If you can be bothered to read the file format specification, open up some files in a hex editor to understand the layout, and write low-level code to parse the data - then you can achieve several orders of magnitude higher performance.

fizx 2y ago

Fun story time: I was at twitter for a few years and tended to write quick hacks that people wanted to replace with better engineering. We never had scala thrift bindings, and the Java ones were awkward from Scala, so I wrote a thrift plugin in JRuby that used the Ruby thrift parser and ERb web templates to output some Scala code. Integrated with our build pipeline, worked great for the company. I also wrote one era of twitter's service deploy system on a hacked up Capistrano. These projects took a few days because they were dirty hacks, but I still got a below perf review for getting easily distracted, because I didn't yet know how to sell those company-wide projects. Anyhow, about a month before that team kicked off Parquet, I showed them a columnar format I made for a hackweek based on Lucene's codec packages, and was using to power a mixpanel-alike analytics system. I'm not sure whether they were inspired or terrified that my hack would reach production, but I like to think I had a small hand in getting Parquet kickstarted.

hi-v-rocknroll 2y ago
Hahaha. Don't ask how the sausage gets made, so long as the sausage machine is readily maintainable by more than one person or has documentation. Partially explains how murder came to be https://github.com/lg/murder https://github.com/lg/murder > below perf review That's some cheap bullshit. Fuck marketing-oriented corporate engineering.

pradeepchhetri 2y ago

Reading through this blog, to me it seems Parquet is lot like ClickHouse native data format. Best part of ClickHouse native data format is I can use the same ClickHouse queries and can run in local or remote server/cluster and let ClickHouse to decide the available resources in the most performant way. ClickHouse has a native and the fastest integration with Parquet so i can: - Query local/s3 parquet data from command line using clickhouse-local. - Query large amount of local/s3 data programmatically by offloading it to clickhouse server/cluster which can do processing in distributed fashion.

pradeepchhetri 2y ago
If you are interested in reading internals of using Parquet with ClickHouse, do read following articles: - https://clickhouse.com/blog/apache-parquet-clickhouse-local-querying-writing https://clickhouse.com/blog/apache-parquet-clickhouse-local-... - https://clickhouse.com/blog/apache-parquet-clickhouse-local-querying-writing-internals-row-groups https://clickhouse.com/blog/apache-parquet-clickhouse-local-...

colloydi 2y ago

Tangential to the topic but regarding the supposed Snowball Effect there is in real life no such thing. I have pushed large 'snowballs' down slopes --in reality they are snow cylinders as shown in the photo-- and they invariably do not get far. The reason being that when one side of the cylinder randomly thickens slightly with respect to the other side this causes the whole thing to turn in the opposite direction. For example, if the RHS of your cylinder has a slightly larger radius than the LHS the cylinder will commence turning to the left. The upshot is the thick side picks up more snow than the thin side and the disparity in radii increases more rapidly still. The cylinder becomes a truncated cone which turns sideways and halts!

whinvik 2y ago
Sounds to me like there's a long blog post waiting to be written.
lsowen 2y ago
It might be rare, but it _can_ happen: https://www.youtube.com/watch?v=ZdIclqha014 https://www.youtube.com/watch?v=ZdIclqha014
dendrite9 2y ago
It is highly dependent on the snow conditions and the recent weather. Sometimes even just the a couple hours are enough to change the conditions to have a good chance of rollerballs. The climate also has an impact, in my experience more coastal areas have more periods when they form. And in some cases the rollerballs get too tall for the bonding strength of the snow, so they break into parts that can restart the cycle if the slope is steep enough.

anentropic 2y ago

Following along the subsequent blog posts in this series progress to Arrow and then to... OpenLineage I'm curious if anyone has experience with OpenLineage/Marquez or similar they'd like to share

Zababa 2y ago

It's very interesting to see how a new "enterprise open source" project is born. The part where right at the start the author knows that they should have more than one company on board, and how the other companies each contribute their part.

heroprotagonist 2y ago

> most basic definition of open source is the code is available for you to read What crap. That's 'source-available', NOT open-source. But at least co-option of terminology is an indicator of success.