4 ms·

Pandas is just the worst. I mean, it's obviously powerful, but it's still the worst. It suffers from the same problem as so many other python libraries which w

Pandas is just the worst. I mean, it's obviously powerful, but it's still the worst. It suffers from the same problem as so many other python libraries which were clearly designed by scientists rather than software engineers. The design philosophy seems to be that having 100s of ways to do every single thing is "convenient". In reality, it makes almost every single function in the pandas library waaaaaaay too complicated. Nearly all of them have different "modes", and sometimes choosing one thing over another will absolutely cripple the performance of your application. I hadn't heard about polars, but I'm currently using pandas in a part of our stack, and since it's not a lot of code yet, I might just be able to replace pandas. Thanks!

deleted 4y ago
[deleted]
d4rkp4ttern 4y ago
Yesterday I spent over 45 mins dealing with this error: https://stackoverflow.com/questions/20625582/how-to-deal-with-settingwithcopywarning-in-pandas https://stackoverflow.com/questions/20625582/how-to-deal-wit... “SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.”
- brahbrah 4y ago
  So in pandas you might get that warning when you’re trying to do an in place operation like so: df.loc['2023-01-01': '2023-12-31'] *= 2 This doubles every row in 2023 in your frame. In polars there’s no mutating operations, everything is pure, so you won’t see any warnings of that nature, but what you will see instead is having to do something like this to solve the same problem: df.with_column([ pl.when(pl.col('date').is_between( datetime(2023, 1, 1), datetime(2023, 12, 31), )).then(pl.col(x) * 2) .otherwise(pl.col(x)) .alias(x) for x in df.columns if x != 'date' ]) So there’s definitely some trade offs, but mutating operations are sometimes worth learning how to use them properly to avoid that pandas warning (or know when to ignore it, because unfortunately there can be false positives).
  - kristianp 4y ago
    When I saw your comment I thought, oh there's another story about polars! Turns out you can reply to a 13 day old thread.

deleted 4y ago

[deleted]

d4rkp4ttern 4y ago

Yesterday I spent over 45 mins dealing with this error: https://stackoverflow.com/questions/20625582/how-to-deal-with-settingwithcopywarning-in-pandas https://stackoverflow.com/questions/20625582/how-to-deal-wit... “SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.”

brahbrah 4y ago
So in pandas you might get that warning when you’re trying to do an in place operation like so: df.loc['2023-01-01': '2023-12-31'] *= 2 This doubles every row in 2023 in your frame. In polars there’s no mutating operations, everything is pure, so you won’t see any warnings of that nature, but what you will see instead is having to do something like this to solve the same problem: df.with_column([ pl.when(pl.col('date').is_between( datetime(2023, 1, 1), datetime(2023, 12, 31), )).then(pl.col(x) * 2) .otherwise(pl.col(x)) .alias(x) for x in df.columns if x != 'date' ]) So there’s definitely some trade offs, but mutating operations are sometimes worth learning how to use them properly to avoid that pandas warning (or know when to ignore it, because unfortunately there can be false positives).
- kristianp 4y ago
  When I saw your comment I thought, oh there's another story about polars! Turns out you can reply to a 13 day old thread.