4 ms·
I know arrow support is only part way their with this release - but this is a huge deal for Pandas, for standardisation as whole, but also speed ups. Benchmark
by benrutter 4y ago
I know arrow support is only part way their with this release - but this is a huge deal for Pandas, for standardisation as whole, but also speed ups.
Benchmarking that was shared a while back here suggests 2x speed ups in some cases, 30x if you count strings since pandas uses python's in-built string data type[1]
[1]https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i https://datapythonista.me/blog/pandas-20-and-the-arrow-revol...
- 0cf8612b2e1e 4y agoNot just speed, but I thought one of the big Arrow wins relative to numpy processing was better memory usage.
- wenc 4y agoPolars is already built on Arrow and the advantages are tremendous.
- benrutter 4y agoYeah Polars is an awesome library! I don't know a whole bunch about its internals, but I think it implements a bunch of additional speed ups through memory allocation and lazy evaluation, so its unlikely Pandas is will get to a similar speed without some huge changes elsewhere
- wenc 4y agoYup here’s a list of optimizations that Polars does to achieve the speed that it does https://pola-rs.github.io/polars-book/user-guide/#current-status https://pola-rs.github.io/polars-book/user-guide/#current-st... Polars’ lazy evaluation is a big deal — this lets it do query plan optimization. Whereas in Pandas every step is eager so it can’t look ahead to eliminate redundant steps. You basically can’t do a lot of query optimization in a multi step transform.