4 ms·
Yeah Polars is an awesome library! I don't know a whole bunch about its internals, but I think it implements a bunch of additional speed ups through memory allo
by benrutter 4y ago
Yeah Polars is an awesome library! I don't know a whole bunch about its internals, but I think it implements a bunch of additional speed ups through memory allocation and lazy evaluation, so its unlikely Pandas is will get to a similar speed without some huge changes elsewhere
- wenc 4y agoYup here’s a list of optimizations that Polars does to achieve the speed that it does https://pola-rs.github.io/polars-book/user-guide/#current-status https://pola-rs.github.io/polars-book/user-guide/#current-st... Polars’ lazy evaluation is a big deal — this lets it do query plan optimization. Whereas in Pandas every step is eager so it can’t look ahead to eliminate redundant steps. You basically can’t do a lot of query optimization in a multi step transform.