2 ms·
If you are working a lot with those parquet files it might be worth looking at Apache Arrow which is an in-memory/wire format for working with columnar data. It
by lawik 3y ago
If you are working a lot with those parquet files it might be worth looking at Apache Arrow which is an in-memory/wire format for working with columnar data. It has a lot of good support for parquet from what I gather and is really focused on allowing efficient wrangling of data. Zero-copy and all that.
Note: no affiliation, just in a deep rabbit hole on data