4 ms·

Anyone got any real-world comparison with Pandas? Like an orders of magnitude wow moment?

minimaxir 3y ago
Real-world performance is complicated since data science covers a lot of use cases. If you're just reading a small CSV to do analysis on it, then there will be no human-perceptible difference between Polars and Pandas. If you're reading a larger CSV with 100k rows, there still won't be much of a perceptible difference. Per this (old) benchmark, there are differences once you get into 10 million rows/500MB+ territory: https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/
- ayhanfuat 3y ago
  DuckDB is publishing updates to the H20.ai benchmark: https://duckdb.org/2023/11/03/db-benchmark-update.html https://duckdb.org/2023/11/03/db-benchmark-update.html
alecst 3y ago
In my experience, the biggest difference is in the API. Besides the fact that it's usually faster than pandas, it also feels faster to write and easier to read.
radus 3y ago
It allowed me to take some code that reads in a bunch of data and performs a few rounds of some pretty standard operations (groupby, filtering, calculating mean/stdevs) and going from pandas to polars allowed me to go from ~1 minute per dataset to 1 second (yes, I tried the Arrow backend for pandas too). This was after spending some time profiling the pandas code and fixing up the slowest parts as best as I could. The translation was pretty straightforward. The output of this pipeline code was a few different dataframes (each to be inserted into a separate table) and each dataframe was output from a function. I was able to migrate one function at a time after asserting that the outputs of the two functions were identical and that all relevant tests passed (I used `to_pandas()` where needed). I'm not sure how much faster I could go, since ~1 second/dataset allowed me to answer some questions that I had that required scanning values for a few parameters. The biggest wins for me were in grouping and merging operations. I'm a complete convert now. The API is simpler and more obvious IMO, and the ability to compose expressions (`polars.Expr`) is awesome. The performance benefits are nice and what motivated me in the first place, but I'm more swayed by the aforementioned benefits.
- mmaunder 3y ago
  Bloody hell!! Thanks, that's exactly the kind of comment I was hoping to see. Sounds like a bit of an Apache --> Nginx moment for dataframes. Super cool!!
  - radus 3y ago
    To add some balance: - I can't rule out that a pandas wizard couldn't have achieved the same speed-up in pandas - polars code was slightly more verbose. For example, when calculating columns based on other columns in the same chain, in pandas, each new column can be defined as a kwarg in a single call to `assign`, whereas in polars, columns that depend on other must be defined in their own calls to `with_columns` - handling of categoricals in polars seemed a little underbaked, though my main complaint, that categories cannot be pre-defined, seems to have been recently addressed: https://github.com/pola-rs/polars/issues/10705 https://github.com/pola-rs/polars/issues/10705 - polars is not yet 1.0, breaking changes will happen
    - billyjmc 3y ago
      Regarding your second point, you can use the walrus operator to retain the results of a computation within a single `.with_columns()` call. See https://stackoverflow.com/a/77609494 https://stackoverflow.com/a/77609494 Edited to add: also, if you’re using a lazy dataframe, you can just naively do the same operation twice (once to store it in a named column and once again in the subsequent computation), and Polars will use common subexpression elimination (CSE) to prevent recomputing the result. You can verify this is true using the `.explain()` method of a lazy dataframe operation containing the `.with_columns()` call.
      - radus 3y ago
        That's awesome, thanks for sharing! Though tbh I'm not likely to use it.. it's a bit too magical - though still a delicious hack.
        3y ago
sanderjd 3y ago
Yep, for some stuff, like anything requiring loading / scanning through all the data (without doing anything requiring a custom lambda), I've seen orders of magnitude improvement to latency, and also (more importantly, imo) much lower memory usage.

Real-world performance is complicated since data science covers a lot of use cases. If you're just reading a small CSV to do analysis on it, then there will be no human-perceptible difference between Polars and Pandas. If you're reading a larger CSV with 100k rows, there still won't be much of a perceptible difference. Per this (old) benchmark, there are differences once you get into 10 million rows/500MB+ territory: https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/

ayhanfuat 3y ago
DuckDB is publishing updates to the H20.ai benchmark: https://duckdb.org/2023/11/03/db-benchmark-update.html https://duckdb.org/2023/11/03/db-benchmark-update.html

alecst 3y ago

In my experience, the biggest difference is in the API. Besides the fact that it's usually faster than pandas, it also feels faster to write and easier to read.

radus 3y ago

It allowed me to take some code that reads in a bunch of data and performs a few rounds of some pretty standard operations (groupby, filtering, calculating mean/stdevs) and going from pandas to polars allowed me to go from ~1 minute per dataset to 1 second (yes, I tried the Arrow backend for pandas too). This was after spending some time profiling the pandas code and fixing up the slowest parts as best as I could. The translation was pretty straightforward. The output of this pipeline code was a few different dataframes (each to be inserted into a separate table) and each dataframe was output from a function. I was able to migrate one function at a time after asserting that the outputs of the two functions were identical and that all relevant tests passed (I used `to_pandas()` where needed). I'm not sure how much faster I could go, since ~1 second/dataset allowed me to answer some questions that I had that required scanning values for a few parameters. The biggest wins for me were in grouping and merging operations. I'm a complete convert now. The API is simpler and more obvious IMO, and the ability to compose expressions (`polars.Expr`) is awesome. The performance benefits are nice and what motivated me in the first place, but I'm more swayed by the aforementioned benefits.

mmaunder 3y ago
Bloody hell!! Thanks, that's exactly the kind of comment I was hoping to see. Sounds like a bit of an Apache --> Nginx moment for dataframes. Super cool!!
- radus 3y ago
  To add some balance: - I can't rule out that a pandas wizard couldn't have achieved the same speed-up in pandas - polars code was slightly more verbose. For example, when calculating columns based on other columns in the same chain, in pandas, each new column can be defined as a kwarg in a single call to `assign`, whereas in polars, columns that depend on other must be defined in their own calls to `with_columns` - handling of categoricals in polars seemed a little underbaked, though my main complaint, that categories cannot be pre-defined, seems to have been recently addressed: https://github.com/pola-rs/polars/issues/10705 https://github.com/pola-rs/polars/issues/10705 - polars is not yet 1.0, breaking changes will happen
  - billyjmc 3y ago
    Regarding your second point, you can use the walrus operator to retain the results of a computation within a single `.with_columns()` call. See https://stackoverflow.com/a/77609494 https://stackoverflow.com/a/77609494 Edited to add: also, if you’re using a lazy dataframe, you can just naively do the same operation twice (once to store it in a named column and once again in the subsequent computation), and Polars will use common subexpression elimination (CSE) to prevent recomputing the result. You can verify this is true using the `.explain()` method of a lazy dataframe operation containing the `.with_columns()` call.
    - radus 3y ago
      That's awesome, thanks for sharing! Though tbh I'm not likely to use it.. it's a bit too magical - though still a delicious hack.
      - 3y ago

sanderjd 3y ago

Yep, for some stuff, like anything requiring loading / scanning through all the data (without doing anything requiring a custom lambda), I've seen orders of magnitude improvement to latency, and also (more importantly, imo) much lower memory usage.