3 ms·
It’ll probably never be fully comparable because Pandas can represent python objects and nulls (badly). However, for the most part Arrow and Numpy are compatib
by sonthonax 6y ago
It’ll probably never be fully comparable because Pandas can represent python objects and nulls (badly). However, for the most part Arrow and Numpy are compatible. There’s no overhead in converting an arrow data structure into a Numpy one.
- derriz 6y agoI don’t think this is the case. Particularly if you move past 1d numpy numeric arrays. And even in the simplest case of say a 1d float32 array, Arrow arrays are chunked which means there is significant overhead if you try to use an arrow table as your data structure when using Python’s scientific/statistics/numerics ecosystem.