4 ms·

I absolutely love Polars, and I use the Rust crate all the time! I think my only gripe is that the Rust API seems (to me at least) to be less well documented t

by p4ul 3y ago

I absolutely love Polars, and I use the Rust crate all the time! I think my only gripe is that the Rust API seems (to me at least) to be less well documented than the Python API. I guess Python is the de facto data science language, so maybe that explains it.

TunaNandos 3y ago
Likewise, I would prefer to stay in Rust when developing command line apps that might do DataFrame thingies and would love more examples of how to transform DataFrames in pure Rust.
laborcontract 3y ago
With GPT4 helping with the refactor, there's no reason to start migrating code away from Pandas imo. A lot of people say they think Pandas is fast enough for their needs, but you're literally getting a 95% speed improvement for free. This is a huge difference in productivity, especially when running code and doing a lot of slicing in notebooks.
- apetresc 3y ago
  I think you meant to say there's no reason not to start migrating? Otherwise I can't parse how the rest of your post matches up with your conclusion.
- twelvechairs 3y ago
  Where ive seen resistance to polars in Python land its been either "Pandas is already used/standard and people understand/know it" or "If you really need speed, you can probably do it in numpy which should be faster again".
- breather 3y ago
  What does GPT-4 have to do with this?
  - laborcontract 3y ago
    It's easier than ever to do a drop-in replacement for data workflows. The whole decision making process between migrating libraries like this is how much time investment is it going to take and how much is it going to pay off?
    - sk11001 3y ago
      Not a good example - polars is new and has changed so much in the past couple of years that GPT-4 often gives outdated code for it.
      - laborcontract 3y ago
        It's still pretty good posting documentation as context or using phind.
- benrutter 3y ago
  Polars is an immense project, and I hope it continues to gain traction. But there's lots more factors than just speed. The main one in my team is ubiquity- i.e. lots of people know pandas, who might not be traditional "developers". I.e. data scientists, data analysts etc. Having a data scientist put together some code, it gets optimized by an engineer, and they can talk back and forth about the same code is a massive benefit. Shifting to polars (and keeping that ability to collaborate) would require not just training the engineers to use a new framework, but all the analysts, data scientists etc that they are adjescant to. That's a huge business cost, and in a lot of cases it might be worth it. But I wouldn't describe it as "getting 95% speed increase for free".
  - laborcontract 3y ago
    While that's fair, it's fairly easy to fit it in only the most intensive operations and then seamlessly convert back to a pandas data frame. I understand why you wouldn’t do this on an organizational level for production workflows, but for personal workflows in my opinion, it’s a no-brainer to incrementally learn and adopt it.
jkrubin 3y ago
I have written several rust/python apis now and I think what you’re feeling is the result of the api being designed for Python as well as being primarily tested in Python. No matter the target, I test rust things using python. idk. Food for thought.
theLiminator 3y ago
Curious why you use the Rust API over the Python API? I'm just wondering if you've also explored the datafusion crate, because I think on the rust side it's about equivalently ergonomic to use either and Datafusion seems a little more modular/pluggable. The Polars python API is definitely miles above the datafusion python API.
- p4ul 3y ago
  Oh, I actually had not yet heard of the DataFusion crate! Thanks for sharing! And to answer your question, we use the Rust API because all of our backend services are in Rust, and we like to stay in Rust whenever possible.

by p4ul 3y ago

TunaNandos 3y ago

Likewise, I would prefer to stay in Rust when developing command line apps that might do DataFrame thingies and would love more examples of how to transform DataFrames in pure Rust.

laborcontract 3y ago

With GPT4 helping with the refactor, there's no reason to start migrating code away from Pandas imo. A lot of people say they think Pandas is fast enough for their needs, but you're literally getting a 95% speed improvement for free. This is a huge difference in productivity, especially when running code and doing a lot of slicing in notebooks.

apetresc 3y ago
I think you meant to say there's no reason not to start migrating? Otherwise I can't parse how the rest of your post matches up with your conclusion.
twelvechairs 3y ago
Where ive seen resistance to polars in Python land its been either "Pandas is already used/standard and people understand/know it" or "If you really need speed, you can probably do it in numpy which should be faster again".
breather 3y ago
What does GPT-4 have to do with this?
- laborcontract 3y ago
  It's easier than ever to do a drop-in replacement for data workflows. The whole decision making process between migrating libraries like this is how much time investment is it going to take and how much is it going to pay off?
  - sk11001 3y ago
    Not a good example - polars is new and has changed so much in the past couple of years that GPT-4 often gives outdated code for it.
    - laborcontract 3y ago
      It's still pretty good posting documentation as context or using phind.
benrutter 3y ago
Polars is an immense project, and I hope it continues to gain traction. But there's lots more factors than just speed. The main one in my team is ubiquity- i.e. lots of people know pandas, who might not be traditional "developers". I.e. data scientists, data analysts etc. Having a data scientist put together some code, it gets optimized by an engineer, and they can talk back and forth about the same code is a massive benefit. Shifting to polars (and keeping that ability to collaborate) would require not just training the engineers to use a new framework, but all the analysts, data scientists etc that they are adjescant to. That's a huge business cost, and in a lot of cases it might be worth it. But I wouldn't describe it as "getting 95% speed increase for free".
- laborcontract 3y ago
  While that's fair, it's fairly easy to fit it in only the most intensive operations and then seamlessly convert back to a pandas data frame. I understand why you wouldn’t do this on an organizational level for production workflows, but for personal workflows in my opinion, it’s a no-brainer to incrementally learn and adopt it.

jkrubin 3y ago

I have written several rust/python apis now and I think what you’re feeling is the result of the api being designed for Python as well as being primarily tested in Python. No matter the target, I test rust things using python. idk. Food for thought.

theLiminator 3y ago

Curious why you use the Rust API over the Python API? I'm just wondering if you've also explored the datafusion crate, because I think on the rust side it's about equivalently ergonomic to use either and Datafusion seems a little more modular/pluggable. The Polars python API is definitely miles above the datafusion python API.

p4ul 3y ago
Oh, I actually had not yet heard of the DataFusion crate! Thanks for sharing! And to answer your question, we use the Rust API because all of our backend services are in Rust, and we like to stay in Rust whenever possible.