5 ms·
You want to search for “DataFrame” libraries. Another commenter mentioned Spark, Panda’s is another popular one, not used it but think it’s lighter weight wher
by BFLpL0QNek 4y ago
You want to search for “DataFrame” libraries.
Another commenter mentioned Spark, Panda’s is another popular one, not used it but think it’s lighter weight where Spark is more for large distributed computation even though can run locally.
There’s a bunch of these tools which lets you treat parquet files as tables doing joins, aggregations etc.
- mountainriver 4y agoArrow is really the future here
- BFLpL0QNek 4y agoIsn’t Apache Arrow an in memory format that the various DataFrame libraries can standardise on to interact with each other? inter-process communication (IPC)? My understanding is your raw data on disk is still a format such as Parquet, but when you load that Parquet in to your application it’s stored as Arrow in-memory for processing?
- hiyer 4y agoArrow also has its own on-disk format called Feather - https://arrow.apache.org/docs/python/feather.html https://arrow.apache.org/docs/python/feather.html