3 ms·
Is that a problem in the Parquet format or in PyArrow? My understanding is that Parquet is primarily meant for on-disk storage (hence the default on-disk compre
by BadInformatics 6y ago
Is that a problem in the Parquet format or in PyArrow? My understanding is that Parquet is primarily meant for on-disk storage (hence the default on-disk compression), so you'd read into Arrow for in-memory compression or IPC.
- IshKebab 6y agoI don't know whether it is a limitation of the format or of the implementation. I used the C++ library (or maybe it was C, I can't remember), which I assume PyArrow uses too. > hence the default on-disk compression No, Parquet doesn't support some compression formats that Arrow does.