2 ms·
Arrow is really the future here
by mountainriver 5y ago
Arrow is really the future here
- BFLpL0QNek 5y agoIsn’t Apache Arrow an in memory format that the various DataFrame libraries can standardise on to interact with each other? inter-process communication (IPC)? My understanding is your raw data on disk is still a format such as Parquet, but when you load that Parquet in to your application it’s stored as Arrow in-memory for processing?
- hiyer 5y agoArrow also has its own on-disk format called Feather - https://arrow.apache.org/docs/python/feather.html https://arrow.apache.org/docs/python/feather.html