4 ms·

I've wanted to convert a massive Pandas codebase to Polars for a long time. Probably 90% of the compute time is Pandas operations, especially creating new colum

I've wanted to convert a massive Pandas codebase to Polars for a long time. Probably 90% of the compute time is Pandas operations, especially creating new columns / resizing dataframes (which I understand to involve less of a speed difference compared to the grouping operations mentioned in the post, but still substantial). Anyone had success doing this and found it to be worth the effort?

wenc 2y ago
I converted to DuckDB and Polars. It’s worth it for the speed improvement. However there are subtle differences between Pandas and Polars behaviors so regression testing is your friend. It’s not 1:1 mapping.
- willseth 2y ago
  Which things did you decide to move to duckdb?
  - wenc 2y ago
    A lot of aggregation, pivot and join logic, like: df = duckdb.query("from tbl1 join tbl2 using (id) where id is not null ").pl() The .pl() is a Arrow-based conversation from DuckDB to Polars. It's in-memory and I believe zero-copy, so it happens almost instantaneously. I go back and forth between DuckDB and Polars functions in the same scope because it's so cheap to convert between the two.
    - __mharrison__ 2y ago
      I'm genuinely curious, if you are already using (and assuming that you like) dataframe APIs, why you would use SQL?
      - wenc 2y ago
        It's the opposite; I prefer DuckDB and generally work with DuckDB's friendly SQL interface. SQL is declarative and is (for me) more intuitive than method-chaining -- especially for complex analytic operations that happen in one go. (software people might beg to differ about the intuitive bit because they are more used to an imperative style, and to my surprise, even the best software engineers struggle with SQL because it requires one to think in set and relation operations rather than function calls, which many software folks are not used to) I actually don't use the Polars dataframe APIs much except for some operations which are easier to do in dataframe form, like applying a Python function as UDF, or transposing (not pivoting) a dataframe. Also Polars is good for materializing the query into a dataframe rapidly, which can then be passed into methods/functions. It's also a lot easier to unit test dataframes than SQL tables. There's a lot more tooling for that.
- kzrdude 2y ago
  There's been so many subtle changes in pandas to pandas upgrades (especially groupby is somehow always hit), so regression tests are always needed...
__mharrison__ 2y ago
It really depends on what you are doing. My book, Effective Polars, has a whole chapter devoted to the question of whether porting from pandas makes sense.
salomonk_mur 2y ago
This is a prime use case for LLMs
- otabdeveloper4 2y ago
  Ah yes, just let the LLM refactor your numeric methods, #yolo. What could go wrong?
- sk11001 2y ago
  It’s a very bad use case for an LLM because they don’t have as much polars code in their training data and the polars API has changed a lot in the past couple of years. Copilot is still suggesting out of date polars code to me. ChatGPT and Claude regularly suggest operations that are valid in pandas and not in polars.
  - sweezyjeezy 2y ago
    Can confirm this, LLMs write almost exclusively in broken polars. However I convert pandas files to polars semi-regularly, and it's still a hell of a lot faster to get an LLM to write the first draft and correct the 4 or 5 attribute errors it makes

wenc 2y ago

I converted to DuckDB and Polars. It’s worth it for the speed improvement. However there are subtle differences between Pandas and Polars behaviors so regression testing is your friend. It’s not 1:1 mapping.

willseth 2y ago
Which things did you decide to move to duckdb?
- wenc 2y ago
  A lot of aggregation, pivot and join logic, like: df = duckdb.query("from tbl1 join tbl2 using (id) where id is not null ").pl() The .pl() is a Arrow-based conversation from DuckDB to Polars. It's in-memory and I believe zero-copy, so it happens almost instantaneously. I go back and forth between DuckDB and Polars functions in the same scope because it's so cheap to convert between the two.
  - __mharrison__ 2y ago
    I'm genuinely curious, if you are already using (and assuming that you like) dataframe APIs, why you would use SQL?
    - wenc 2y ago
      It's the opposite; I prefer DuckDB and generally work with DuckDB's friendly SQL interface. SQL is declarative and is (for me) more intuitive than method-chaining -- especially for complex analytic operations that happen in one go. (software people might beg to differ about the intuitive bit because they are more used to an imperative style, and to my surprise, even the best software engineers struggle with SQL because it requires one to think in set and relation operations rather than function calls, which many software folks are not used to) I actually don't use the Polars dataframe APIs much except for some operations which are easier to do in dataframe form, like applying a Python function as UDF, or transposing (not pivoting) a dataframe. Also Polars is good for materializing the query into a dataframe rapidly, which can then be passed into methods/functions. It's also a lot easier to unit test dataframes than SQL tables. There's a lot more tooling for that.
kzrdude 2y ago
There's been so many subtle changes in pandas to pandas upgrades (especially groupby is somehow always hit), so regression tests are always needed...

__mharrison__ 2y ago

It really depends on what you are doing. My book, Effective Polars, has a whole chapter devoted to the question of whether porting from pandas makes sense.

salomonk_mur 2y ago

This is a prime use case for LLMs

otabdeveloper4 2y ago
Ah yes, just let the LLM refactor your numeric methods, #yolo. What could go wrong?
sk11001 2y ago
It’s a very bad use case for an LLM because they don’t have as much polars code in their training data and the polars API has changed a lot in the past couple of years. Copilot is still suggesting out of date polars code to me. ChatGPT and Claude regularly suggest operations that are valid in pandas and not in polars.
- sweezyjeezy 2y ago
  Can confirm this, LLMs write almost exclusively in broken polars. However I convert pandas files to polars semi-regularly, and it's still a hell of a lot faster to get an LLM to write the first draft and correct the 4 or 5 attribute errors it makes