5 ms·

Why is it not in Debian? Is there some deep and dark secret? https://search.debian.org/cgi-bin/omega?DB=en&P=parquet https://search.debian.org/cgi-bin/omega?D

Why is it not in Debian? Is there some deep and dark secret? https://search.debian.org/cgi-bin/omega?DB=en&P=parquet https://search.debian.org/cgi-bin/omega?DB=en&P=parquet

whinvik 2y ago
Parquet is a file format. Should a file format be in Debian?
- jjgreen 2y ago
  I mean library support for reading and writing it: > apt-cache search hdf5 | wc -l 134 > apt-cache search netcdf | wc -l 70 > apt-cache search parquet | wc -l 0
  - jjtheblunt 2y ago
    look for polars, https://pola.rs https://pola.rs
    - jjgreen 2y ago
      https://packages.debian.org/search?searchon=names&keywords=polars https://packages.debian.org/search?searchon=names&keywords=p...
      - petre 2y ago
        Use pip, Luke! https://docs.pola.rs/user-guide/getting-started https://docs.pola.rs/user-guide/getting-started
        jjgreen 2y ago
        My question is "why isn't it in Debian?", I ask that since Debian has rather high standards and the absence from Debian suggests some quality issue in available libraries for the format or the format itself. Are there dark secrets?
        jjtheblunt 2y ago
        Could the dark secret you seek be "debian isn't for bleeding edge packages"? it's very modern and perhaps hasn't been around long enough to have debian maintainers feel it's vetted. for instance, documentation for Python bindings is more advanced than for Rust bindings, but the package itself uses Rust at the low level.
        jjgreen 2y ago
        Parquet is what, 12 years old? Hardly cutting edge. What you say my well be true for polars (I'm not familiar with it), if/when it (or something else) does get packaged I'll give parquet another look ...
        jjtheblunt 2y ago
        Yes, i wasn't clear: it's the polars library that's actively changing, so that might be the issue, or just the vast set of optional components configurable on installation, which isn't the normal package manager experience. FWIW i think i share your general aversion to _not_ using packages, just for the tidiness of installs and removals, though i'm on fedora and macos.
        petre 2y ago
        Pandas is probably in Debian and it can read parquet files. Polars is fairly new and under active development. It's a python library, I install those in $HOME/.local, as opposed to system wide. One can also install it in a venv. With pip you can also uninstall packages and keep things fairly tidy.
        jjgreen 2y ago
        Pandas is in Debian but it cannot read parquet files itself, it uses 3rd party "engines" for that purpose and those are not available in Debian Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pandas >>> pandas.read_parquet('sample3.parquet') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3/dist-packages/pandas/io/parquet.py", line 493, in read_parquet impl = get_engine(engine) File "/usr/lib/python3/dist-packages/pandas/io/parquet.py", line 53, in get_engine raise ImportError( ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'. A suitable version of pyarrow or fastparquet is required for parquet support.
      - jjtheblunt 2y ago
        Why did you ignore the link i gave you above? If you follow that link, you'll see polars and parquet are a large highly configurable collection of tools for format manipulations across many HPC formats. Debian maintainers possibly don't want to bundle the entirety, as it would be vast. Might this help you, though? https://cloudsmith.io/~opencpn/repos/polar-prod/packages/detail/raw/polar_pi-1.2.6.0-debian-x86_64-10-buster-tarball/v1.2.6.0/ https://cloudsmith.io/~opencpn/repos/polar-prod/packages/det...
  - whinvik 2y ago
    Are there Arrow libraries. Feel like a lot of applications that read Parquet actually outsource raw reads to Arrow.
  - marginalia_nu 2y ago
    Duckdb is probably what you want, though I don't think it's in debian either. It's in Arch though.
mistrial9 2y ago
pandas is a python-centric, tabular data handler that works well in clouds (and desktop Debian). Pandas can read parquet data today, among other libs mentioned. The binary dot-so driver style is single-host centric and not the emphasis of these cloudy projects (and their cloudy funders) https://pandas.pydata.org/docs/reference/api/pandas.read_parquet.html https://pandas.pydata.org/docs/reference/api/pandas.read_par... https://packages.debian.org/buster/python3-pandas https://packages.debian.org/buster/python3-pandas Perhaps more alarm is called for when this python+pandas and parquet does not work on Debian, but that is not the case today. ps- data access in clouds often uses the S3:// endpoint . Contrast to a POSIX endpoint using _fread()_ or similar.. many parquet-aware clients prefer the cloudy, un-POSIX method to access data and that is another reason it is not a simple package in Debian today.
- jjgreen 2y ago
  As I understand it, pandas can read parquet if the pyarrow or fastparquet packages are available, but that's not the case and attempts to fix that have been underway for several years. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021
  - mistrial9 2y ago
    hm https://github.com/pandas-dev/pandas/blob/main/pandas/io/parquet.py https://github.com/pandas-dev/pandas/blob/main/pandas/io/par... says "fastparquet" engine must be available if no pyarrow https://pypi.org/project/fastparquet/ https://pypi.org/project/fastparquet/
    - jjgreen 2y ago
      See https://news.ycombinator.com/item?id=40306445 https://news.ycombinator.com/item?id=40306445
- datadrivenangel 2y ago
  Pandas often has significant memory overhead, so it's not uncommon to need ~3-5x the amount of memory as your file size. Polars and DuckDB are much better about memory management.
pabs3 2y ago
No-one with the combination of motivation, time and skills needed get it into Debian. Someone wanted to get a Python implementation in, but it looks like they never found the time. https://bugs.debian.org/838338 https://bugs.debian.org/838338 These days Debian packaging has become a bit irrelevant, since you can just shove upstream releases into a container and go for it.
- mkesper 2y ago
  If you want to depend on curl|bash setups, sure. If you want to get security updates, good packaging is still required.

Why is it not in Debian? Is there some deep and dark secret? https://search.debian.org/cgi-bin/omega?DB=en&P=parquet https://search.debian.org/cgi-bin/omega?DB=en&P=parquet

whinvik 2y ago

Parquet is a file format. Should a file format be in Debian?

jjgreen 2y ago
I mean library support for reading and writing it: > apt-cache search hdf5 | wc -l 134 > apt-cache search netcdf | wc -l 70 > apt-cache search parquet | wc -l 0
- jjtheblunt 2y ago
  look for polars, https://pola.rs https://pola.rs
  - jjgreen 2y ago
    https://packages.debian.org/search?searchon=names&keywords=polars https://packages.debian.org/search?searchon=names&keywords=p...
    - petre 2y ago
      Use pip, Luke! https://docs.pola.rs/user-guide/getting-started https://docs.pola.rs/user-guide/getting-started
      - jjgreen 2y ago
        My question is "why isn't it in Debian?", I ask that since Debian has rather high standards and the absence from Debian suggests some quality issue in available libraries for the format or the format itself. Are there dark secrets?
        jjtheblunt 2y ago
        Could the dark secret you seek be "debian isn't for bleeding edge packages"? it's very modern and perhaps hasn't been around long enough to have debian maintainers feel it's vetted. for instance, documentation for Python bindings is more advanced than for Rust bindings, but the package itself uses Rust at the low level.
        jjgreen 2y ago
        Parquet is what, 12 years old? Hardly cutting edge. What you say my well be true for polars (I'm not familiar with it), if/when it (or something else) does get packaged I'll give parquet another look ...
        jjtheblunt 2y ago
        Yes, i wasn't clear: it's the polars library that's actively changing, so that might be the issue, or just the vast set of optional components configurable on installation, which isn't the normal package manager experience. FWIW i think i share your general aversion to _not_ using packages, just for the tidiness of installs and removals, though i'm on fedora and macos.
        petre 2y ago
        Pandas is probably in Debian and it can read parquet files. Polars is fairly new and under active development. It's a python library, I install those in $HOME/.local, as opposed to system wide. One can also install it in a venv. With pip you can also uninstall packages and keep things fairly tidy.
        jjgreen 2y ago
        Pandas is in Debian but it cannot read parquet files itself, it uses 3rd party "engines" for that purpose and those are not available in Debian Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pandas >>> pandas.read_parquet('sample3.parquet') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3/dist-packages/pandas/io/parquet.py", line 493, in read_parquet impl = get_engine(engine) File "/usr/lib/python3/dist-packages/pandas/io/parquet.py", line 53, in get_engine raise ImportError( ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'. A suitable version of pyarrow or fastparquet is required for parquet support.
    - jjtheblunt 2y ago
      Why did you ignore the link i gave you above? If you follow that link, you'll see polars and parquet are a large highly configurable collection of tools for format manipulations across many HPC formats. Debian maintainers possibly don't want to bundle the entirety, as it would be vast. Might this help you, though? https://cloudsmith.io/~opencpn/repos/polar-prod/packages/detail/raw/polar_pi-1.2.6.0-debian-x86_64-10-buster-tarball/v1.2.6.0/ https://cloudsmith.io/~opencpn/repos/polar-prod/packages/det...
- whinvik 2y ago
  Are there Arrow libraries. Feel like a lot of applications that read Parquet actually outsource raw reads to Arrow.
- marginalia_nu 2y ago
  Duckdb is probably what you want, though I don't think it's in debian either. It's in Arch though.

mistrial9 2y ago

pandas is a python-centric, tabular data handler that works well in clouds (and desktop Debian). Pandas can read parquet data today, among other libs mentioned. The binary dot-so driver style is single-host centric and not the emphasis of these cloudy projects (and their cloudy funders) https://pandas.pydata.org/docs/reference/api/pandas.read_parquet.html https://pandas.pydata.org/docs/reference/api/pandas.read_par... https://packages.debian.org/buster/python3-pandas https://packages.debian.org/buster/python3-pandas Perhaps more alarm is called for when this python+pandas and parquet does not work on Debian, but that is not the case today. ps- data access in clouds often uses the S3:// endpoint . Contrast to a POSIX endpoint using _fread()_ or similar.. many parquet-aware clients prefer the cloudy, un-POSIX method to access data and that is another reason it is not a simple package in Debian today.

jjgreen 2y ago
As I understand it, pandas can read parquet if the pyarrow or fastparquet packages are available, but that's not the case and attempts to fix that have been underway for several years. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021
- mistrial9 2y ago
  hm https://github.com/pandas-dev/pandas/blob/main/pandas/io/parquet.py https://github.com/pandas-dev/pandas/blob/main/pandas/io/par... says "fastparquet" engine must be available if no pyarrow https://pypi.org/project/fastparquet/ https://pypi.org/project/fastparquet/
  - jjgreen 2y ago
    See https://news.ycombinator.com/item?id=40306445 https://news.ycombinator.com/item?id=40306445
datadrivenangel 2y ago
Pandas often has significant memory overhead, so it's not uncommon to need ~3-5x the amount of memory as your file size. Polars and DuckDB are much better about memory management.

pabs3 2y ago

No-one with the combination of motivation, time and skills needed get it into Debian. Someone wanted to get a Python implementation in, but it looks like they never found the time. https://bugs.debian.org/838338 https://bugs.debian.org/838338 These days Debian packaging has become a bit irrelevant, since you can just shove upstream releases into a container and go for it.

mkesper 2y ago
If you want to depend on curl|bash setups, sure. If you want to get security updates, good packaging is still required.