4 ms·

"Polars is based on the Rust native implementation Apache Arrow. Arrow can be seen as middleware software for DBMS, query engines and DataFrame libraries. Arrow

"Polars is based on the Rust native implementation Apache Arrow. Arrow can be seen as middleware software for DBMS, query engines and DataFrame libraries. Arrow provides very cache-coherent data structures and proper missing data handling." This is super cool. Anyone know if Pandas is also planning to adopt Arrow ?

ellimilial 6y ago
It's on their roadmap https://pandas.pydata.org/docs/development/roadmap.html#apache-arrow-interoperability https://pandas.pydata.org/docs/development/roadmap.html#apac...
SatvikBeri 6y ago
I believe Pandas is incompatible with Arrow for a few reasons, such as their indexes and datetime types. But it's pretty easy to convert a pandas dataframe to Arrow and vice versa – I actually use this to pass data between Python & Julia. As a side note, Wes McKinney, the creator of Pandas, is heavily involved in Arrow.
sonthonax 6y ago
It’ll probably never be fully comparable because Pandas can represent python objects and nulls (badly). However, for the most part Arrow and Numpy are compatible. There’s no overhead in converting an arrow data structure into a Numpy one.
- derriz 6y ago
  I don’t think this is the case. Particularly if you move past 1d numpy numeric arrays. And even in the simplest case of say a 1d float32 array, Arrow arrays are chunked which means there is significant overhead if you try to use an arrow table as your data structure when using Python’s scientific/statistics/numerics ecosystem.
sterlinm 6y ago
I could be misremembering but I seem to recall Wes McKinney saying in some talk that rather than rewrite Pandas to be Arrow-backed it will probably eventually be replaced by newer Arrow-backed libraries some of which might have pandas-like apis. I think the idea was that pandas API is too large and the library too widely used for it to be practical to correct some of the design problems people have mentioned. He'd sketched out a vision for Pandas 2.0 at one point and I think he said that basically that would probably just be a new library. There's a lot of related discussion in this post on his blog. https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/

ellimilial 6y ago

It's on their roadmap https://pandas.pydata.org/docs/development/roadmap.html#apache-arrow-interoperability https://pandas.pydata.org/docs/development/roadmap.html#apac...

SatvikBeri 6y ago

I believe Pandas is incompatible with Arrow for a few reasons, such as their indexes and datetime types. But it's pretty easy to convert a pandas dataframe to Arrow and vice versa – I actually use this to pass data between Python & Julia. As a side note, Wes McKinney, the creator of Pandas, is heavily involved in Arrow.

sonthonax 6y ago

It’ll probably never be fully comparable because Pandas can represent python objects and nulls (badly). However, for the most part Arrow and Numpy are compatible. There’s no overhead in converting an arrow data structure into a Numpy one.

derriz 6y ago
I don’t think this is the case. Particularly if you move past 1d numpy numeric arrays. And even in the simplest case of say a 1d float32 array, Arrow arrays are chunked which means there is significant overhead if you try to use an arrow table as your data structure when using Python’s scientific/statistics/numerics ecosystem.

sterlinm 6y ago

I could be misremembering but I seem to recall Wes McKinney saying in some talk that rather than rewrite Pandas to be Arrow-backed it will probably eventually be replaced by newer Arrow-backed libraries some of which might have pandas-like apis. I think the idea was that pandas API is too large and the library too widely used for it to be practical to correct some of the design problems people have mentioned. He'd sketched out a vision for Pandas 2.0 at one point and I think he said that basically that would probably just be a new library. There's a lot of related discussion in this post on his blog. https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/