4 ms·
Isn’t Apache Arrow an in memory format that the various DataFrame libraries can standardise on to interact with each other? inter-process communication (IPC)?
by BFLpL0QNek 4y ago
Isn’t Apache Arrow an in memory format that the various DataFrame libraries can standardise on to interact with each other? inter-process communication (IPC)?
My understanding is your raw data on disk is still a format such as Parquet, but when you load that Parquet in to your application it’s stored as Arrow in-memory for processing?
- hiyer 4y agoArrow also has its own on-disk format called Feather - https://arrow.apache.org/docs/python/feather.html https://arrow.apache.org/docs/python/feather.html