4 ms·

I've migrated off of pandas to polars for my workflows to reap the benefit of, in my experience a 10-20x speedup on average. I can't imagine anything bringing m

I've migrated off of pandas to polars for my workflows to reap the benefit of, in my experience a 10-20x speedup on average. I can't imagine anything bringing me back short of a performance miracle. LLMs have made syntax almost a non-barrier.

alex7o 8mo ago
Same, also polars works on typescript which I used at some point out move my data from backend to frontend
gHA5 8mo ago
Do you not experience LLM generated code constantly trying to use Pandas' methods/syntax for Polars objects?
- postalcoder 8mo ago
  There were some growing pains in gpt-3.5 to gpt-4 era, but not nowadays (shoutout to the now-defunct Phind, which was a game changer back then).
  - crimsoneer 8mo ago
    The fact they pivoted away from their very compelling core offering (AI stack overflow) to complete with loveable etc in the "AI generated apps" giant fight continues to baffle me. Though I guess model updates ate their lunch.
    - postalcoder 8mo ago
      My guess is that their pivot came after distress, and was not the cause of it. It'd be great to have @rushingcreek write a post-mortem. I think it'd benefit a lot of people because I honestly don't have a monday morning playbook of what could have saved them. Like you said, perhaps the demise of phind was inevitable, with large models displacing them kind of like how Spotify displaced music piracy.
- edschofield 8mo ago
  Yes, ChatGPT 5.2 Pro absolutely still does this. Just ask it for a pivot table using Polars and it will probably spit out code with Pandas arguments that doesn’t work.
  - deleted 8mo ago
    [deleted]
mritchie712 8mo ago
also migrated, but to duckdb. It's funny to look back at the tricks that were needed to get gpt3 and 3.5 to write SQL (e.g. "you are a data analyst looking at a SQL database with table [tables]"). It's almost effortless now.
- wodenokoto 8mo ago
  Do you use it from within Python or just ingest straight into duckdb.exe or duckdb UI?
howling 8mo ago
Same. I don't even use LLM normally as I found polars' syntax to be very intuitive. I just searched my ChatGPT history and the only times I used it are when I'm dealing with list and struct columns that were not in pandas.
- postalcoder 8mo ago
  iirc part of pandas’ popularity was that it modeled some of R’s ergonomics. What a time in history, when such things mattered! (To be clear, I’m not making fun of pandas. It was the bridge I crossed that moved me from living in Excel to living in code.)
  - iugtmkbdfil834 8mo ago
    I learned about pandas with R in my class way back when. At the time, it seemed like magic. In a sense, it still does, but things evolve.
OutOfHere 8mo ago
The speedup you claim is going to be contingent on how you use Pandas, with which data types, and which version of Pandas.
lvl155 8mo ago
Went from pandas to polars to duckdb. As mentioned elsewhere SQL is the most readable for me and LLM does most of the coding on my end (quant). So I need it at the most readable and rudimentary/step-wise level. OT, but I can’t imagine data science being a job category for too long. It’s got to be one of the first to go in AI age especially since the market is so saturated with mediocre talents.
- iugtmkbdfil834 8mo ago
  << It’s got to be one of the first to go in AI age especially since the market is so saturated with mediocre talents. This is interesting. I wanted to dig into it a little since I am not sure I am following the logic of that statement. Do you mean that AI would take over the field, because by default most people there are already not producing anything that a simple 'talk to data' LLM won't deliver?
  - mynameisash 8mo ago
    Not GP, but as a data engineer who has worked with data scientists for 20 years, I think the assessment is unfortunately true. I used to work on teams where DS would put a ton of time into building quality models, gating production with defensible metrics. Now, my DS counterparts are writing prompts and calling it a day. I'm not at all convinced that the results are better, but I guess if you don't spend time (=money) on the work, it's hard to argue with the ROI?
    - datsci_est_2015 8mo ago
      In what field do you work? > writing prompts and calling it a day What does this mean? They’re not creating pull requests and maintaining learning / analytics systems? This kind of vagueposting gets on my nerves.
      - mynameisash 8mo ago
        > They’re not creating pull requests and maintaining learning / analytics systems? Sure, they check prompts into git. And there are a few notebooks that have been written and deployed, but most of that is collecting data and handing it off to ChatGPT. No, they're not maintaining learning/analytics systems. My team builds our data processing pipelines, and we support everything in production. > This kind of vagueposting gets on my nerves. What is vague about my comment? Whereas in the past, the DS teams I worked with would do feature engineering and rigorous evaluation of models with retraining based on different criteria, now I'm seeing that teams are being lazy and saying, "We'll let the LLM do things. It can handle unstructured data, and we can give it new data without additional work on our part." Hence, they're simply writing a prompt and not doing much more.
thibaut_barrere 8mo ago
Polars being so fast, and embeddable into other languages, has made it a no brainer for me to adopt it. I have integrated Explorer https://github.com/elixir-explorer/explorer https://github.com/elixir-explorer/explorer, which leverages it, into many Elixir apps, so happy to have this.
- deleted 8mo ago
  [deleted]
thegabriele 8mo ago
" 10-20x speedup on average. " Is this everyone's experience?
- mynameisash 8mo ago
  That was probably about what I got when I migrated some heavy number crunching code from Pandas to Polars a few years ago. Maybe even better than that.
- OGWhales 8mo ago
  It depends on the specifics, but I converted a couple of scripts recently that would take minutes to run with Pandas that only took seconds to run with Polars. I was pretty impressed.
- mjhay 8mo ago
  It’s a typical experience. Polars is fast, and Pandas is very slow and memory-hungry. It would be one thing if Pandas had a good API, but it doesn’t.

alex7o 8mo ago

Same, also polars works on typescript which I used at some point out move my data from backend to frontend

gHA5 8mo ago

Do you not experience LLM generated code constantly trying to use Pandas' methods/syntax for Polars objects?

postalcoder 8mo ago
There were some growing pains in gpt-3.5 to gpt-4 era, but not nowadays (shoutout to the now-defunct Phind, which was a game changer back then).
- crimsoneer 8mo ago
  The fact they pivoted away from their very compelling core offering (AI stack overflow) to complete with loveable etc in the "AI generated apps" giant fight continues to baffle me. Though I guess model updates ate their lunch.
  - postalcoder 8mo ago
    My guess is that their pivot came after distress, and was not the cause of it. It'd be great to have @rushingcreek write a post-mortem. I think it'd benefit a lot of people because I honestly don't have a monday morning playbook of what could have saved them. Like you said, perhaps the demise of phind was inevitable, with large models displacing them kind of like how Spotify displaced music piracy.
edschofield 8mo ago
Yes, ChatGPT 5.2 Pro absolutely still does this. Just ask it for a pivot table using Polars and it will probably spit out code with Pandas arguments that doesn’t work.
- deleted 8mo ago
  [deleted]

mritchie712 8mo ago

also migrated, but to duckdb. It's funny to look back at the tricks that were needed to get gpt3 and 3.5 to write SQL (e.g. "you are a data analyst looking at a SQL database with table [tables]"). It's almost effortless now.

wodenokoto 8mo ago
Do you use it from within Python or just ingest straight into duckdb.exe or duckdb UI?

howling 8mo ago

Same. I don't even use LLM normally as I found polars' syntax to be very intuitive. I just searched my ChatGPT history and the only times I used it are when I'm dealing with list and struct columns that were not in pandas.

postalcoder 8mo ago
iirc part of pandas’ popularity was that it modeled some of R’s ergonomics. What a time in history, when such things mattered! (To be clear, I’m not making fun of pandas. It was the bridge I crossed that moved me from living in Excel to living in code.)
- iugtmkbdfil834 8mo ago
  I learned about pandas with R in my class way back when. At the time, it seemed like magic. In a sense, it still does, but things evolve.

OutOfHere 8mo ago

The speedup you claim is going to be contingent on how you use Pandas, with which data types, and which version of Pandas.

lvl155 8mo ago

Went from pandas to polars to duckdb. As mentioned elsewhere SQL is the most readable for me and LLM does most of the coding on my end (quant). So I need it at the most readable and rudimentary/step-wise level. OT, but I can’t imagine data science being a job category for too long. It’s got to be one of the first to go in AI age especially since the market is so saturated with mediocre talents.

iugtmkbdfil834 8mo ago
<< It’s got to be one of the first to go in AI age especially since the market is so saturated with mediocre talents. This is interesting. I wanted to dig into it a little since I am not sure I am following the logic of that statement. Do you mean that AI would take over the field, because by default most people there are already not producing anything that a simple 'talk to data' LLM won't deliver?
- mynameisash 8mo ago
  Not GP, but as a data engineer who has worked with data scientists for 20 years, I think the assessment is unfortunately true. I used to work on teams where DS would put a ton of time into building quality models, gating production with defensible metrics. Now, my DS counterparts are writing prompts and calling it a day. I'm not at all convinced that the results are better, but I guess if you don't spend time (=money) on the work, it's hard to argue with the ROI?
  - datsci_est_2015 8mo ago
    In what field do you work? > writing prompts and calling it a day What does this mean? They’re not creating pull requests and maintaining learning / analytics systems? This kind of vagueposting gets on my nerves.
    - mynameisash 8mo ago
      > They’re not creating pull requests and maintaining learning / analytics systems? Sure, they check prompts into git. And there are a few notebooks that have been written and deployed, but most of that is collecting data and handing it off to ChatGPT. No, they're not maintaining learning/analytics systems. My team builds our data processing pipelines, and we support everything in production. > This kind of vagueposting gets on my nerves. What is vague about my comment? Whereas in the past, the DS teams I worked with would do feature engineering and rigorous evaluation of models with retraining based on different criteria, now I'm seeing that teams are being lazy and saying, "We'll let the LLM do things. It can handle unstructured data, and we can give it new data without additional work on our part." Hence, they're simply writing a prompt and not doing much more.

thibaut_barrere 8mo ago

Polars being so fast, and embeddable into other languages, has made it a no brainer for me to adopt it. I have integrated Explorer https://github.com/elixir-explorer/explorer https://github.com/elixir-explorer/explorer, which leverages it, into many Elixir apps, so happy to have this.

deleted 8mo ago
[deleted]

thegabriele 8mo ago

" 10-20x speedup on average. " Is this everyone's experience?

mynameisash 8mo ago
That was probably about what I got when I migrated some heavy number crunching code from Pandas to Polars a few years ago. Maybe even better than that.
OGWhales 8mo ago
It depends on the specifics, but I converted a couple of scripts recently that would take minutes to run with Pandas that only took seconds to run with Polars. I was pretty impressed.
mjhay 8mo ago
It’s a typical experience. Polars is fast, and Pandas is very slow and memory-hungry. It would be one thing if Pandas had a good API, but it doesn’t.