20 ms·
Yeah I do. For example Apache Arrow supports in memory compression. But Parquet does not support that. I had to look through the code to find that out, and I fo
by IshKebab 6y ago
Yeah I do. For example Apache Arrow supports in memory compression. But Parquet does not support that. I had to look through the code to find that out, and I found many instances of basically `throw "Not supported"`. And yeah as I said the documentation is just non-existent.
If you are already using Arrow, or you absolutely must use a columnar file format then it's probably a good option.
- BadInformatics 6y agoIs that a problem in the Parquet format or in PyArrow? My understanding is that Parquet is primarily meant for on-disk storage (hence the default on-disk compression), so you'd read into Arrow for in-memory compression or IPC.
- IshKebab 6y agoI don't know whether it is a limitation of the format or of the implementation. I used the C++ library (or maybe it was C, I can't remember), which I assume PyArrow uses too. > hence the default on-disk compression No, Parquet doesn't support some compression formats that Arrow does.