3 ms·

pandas is a python-centric, tabular data handler that works well in clouds (and desktop Debian). Pandas can read parquet data today, among other libs mentioned.

pandas is a python-centric, tabular data handler that works well in clouds (and desktop Debian). Pandas can read parquet data today, among other libs mentioned. The binary dot-so driver style is single-host centric and not the emphasis of these cloudy projects (and their cloudy funders) https://pandas.pydata.org/docs/reference/api/pandas.read_parquet.html https://pandas.pydata.org/docs/reference/api/pandas.read_par... https://packages.debian.org/buster/python3-pandas https://packages.debian.org/buster/python3-pandas Perhaps more alarm is called for when this python+pandas and parquet does not work on Debian, but that is not the case today. ps- data access in clouds often uses the S3:// endpoint . Contrast to a POSIX endpoint using _fread()_ or similar.. many parquet-aware clients prefer the cloudy, un-POSIX method to access data and that is another reason it is not a simple package in Debian today.

jjgreen 2y ago
As I understand it, pandas can read parquet if the pyarrow or fastparquet packages are available, but that's not the case and attempts to fix that have been underway for several years. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021
- mistrial9 2y ago
  hm https://github.com/pandas-dev/pandas/blob/main/pandas/io/parquet.py https://github.com/pandas-dev/pandas/blob/main/pandas/io/par... says "fastparquet" engine must be available if no pyarrow https://pypi.org/project/fastparquet/ https://pypi.org/project/fastparquet/
  - jjgreen 2y ago
    See https://news.ycombinator.com/item?id=40306445 https://news.ycombinator.com/item?id=40306445
datadrivenangel 2y ago
Pandas often has significant memory overhead, so it's not uncommon to need ~3-5x the amount of memory as your file size. Polars and DuckDB are much better about memory management.
- marton78 2y ago
  Also, Polars has a sane and well thought out API, unlike the hot streaming mess that is Pandas.

jjgreen 2y ago

As I understand it, pandas can read parquet if the pyarrow or fastparquet packages are available, but that's not the case and attempts to fix that have been underway for several years. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021

mistrial9 2y ago
hm https://github.com/pandas-dev/pandas/blob/main/pandas/io/parquet.py https://github.com/pandas-dev/pandas/blob/main/pandas/io/par... says "fastparquet" engine must be available if no pyarrow https://pypi.org/project/fastparquet/ https://pypi.org/project/fastparquet/
- jjgreen 2y ago
  See https://news.ycombinator.com/item?id=40306445 https://news.ycombinator.com/item?id=40306445

datadrivenangel 2y ago

Pandas often has significant memory overhead, so it's not uncommon to need ~3-5x the amount of memory as your file size. Polars and DuckDB are much better about memory management.

marton78 2y ago
Also, Polars has a sane and well thought out API, unlike the hot streaming mess that is Pandas.