6 ms·

Dope. I've just started using Pandas in some personal projects, and am quickly hitting my knowledge ceiling. I think this will be useful. I'll check it out prop

by rithdmc 7mo ago

Dope. I've just started using Pandas in some personal projects, and am quickly hitting my knowledge ceiling. I think this will be useful. I'll check it out properly after work.

data-ottawa 7mo ago
You should check out the Modern Pandas series by Tom Augspurger, it’s well worth reading to get clean modern style code. https://tomaugspurger.net/posts/modern-1-intro/ https://tomaugspurger.net/posts/modern-1-intro/
- pixelispoint 7mo ago
  I second this blog post. I worked with Tom on a project several years ago and he's brilliant. Started doing python more frequently after that project and I found his blog to be very helpful in finding a good way to conceptualize pandas and python data structures in general.
- rithdmc 7mo ago
  Thanks. There's a special place in my heart for any blog that opens with 'Prior Work' :)
- sceadu 7mo ago
  also I would recommend looking at videos from matt harrison for polars or pandas, e.g.: https://www.youtube.com/watch?v=Z9ekw2Ou3s0 https://www.youtube.com/watch?v=Z9ekw2Ou3s0
derriz 7mo ago
If I were investing effort into acquiring knowledge in this domain, I'd skip straight to Polars. Before I made the switch, I had been using Pandas on and off for more than a decade. I'm not sure how representative this is, but most of the people I know who were Pandas users have also made this switch. I initially did it for the performance improvements but the API (according to my subjective opinion) is much more logical and has far fewer surprises compared to Pandas and it would be my default choice for this reason alone at this stage despite my years of Pandas experience.
- rithdmc 7mo ago
  Thanks, I'll look into this in the future. I don't need the most performant script, but this could change.
  - ertgbnm 7mo ago
    It's less about performance and more about ecosystem lockin. It's a bit like imperial vs metric units. Why would you ever chose to learn imperial if you had the option to only ever use metric to begin with?
    - rithdmc 7mo ago
      Because these are silly personal scripts. I'm not going to make sensible architectural decisions on something I run every now and then on my laptop. That's optimising too early.
    - short_sells_poo 7mo ago
      For short scripts and interactive research work, pandas is still much better than polars. Polars works well when you know what you want. When you are still figuring out things step by step, pandas does a lot of heavy lifting for you so you don't have to think about it. E.g. I don't have to think about timeseries alignment, pandas handles that for me implicitly because dataframes can be indexed by timestamps. Polars has timeseries support, but I need to write a paragraph of extra code to deal with it.
    - dec0dedab0de 7mo ago
      That's exactly why I am reluctant to do anything with Polars. They are actively running a company and trying to sell a product. At any point they could be acquired and change the license for new releases. Sure you could fork it, or stay on an older version, but if what they offer isn't compelling enough for you then why take the risk? Pandas on the other hand has been open source for almost two decades, and is supported by many companies. They have a governance board, and an active community. The risk of it going off the rails into corporate nonsense is much lower.

by rithdmc 7mo ago

Dope. I've just started using Pandas in some personal projects, and am quickly hitting my knowledge ceiling. I think this will be useful. I'll check it out properly after work.

data-ottawa 7mo ago

You should check out the Modern Pandas series by Tom Augspurger, it’s well worth reading to get clean modern style code. https://tomaugspurger.net/posts/modern-1-intro/ https://tomaugspurger.net/posts/modern-1-intro/

pixelispoint 7mo ago
I second this blog post. I worked with Tom on a project several years ago and he's brilliant. Started doing python more frequently after that project and I found his blog to be very helpful in finding a good way to conceptualize pandas and python data structures in general.
rithdmc 7mo ago
Thanks. There's a special place in my heart for any blog that opens with 'Prior Work' :)
sceadu 7mo ago
also I would recommend looking at videos from matt harrison for polars or pandas, e.g.: https://www.youtube.com/watch?v=Z9ekw2Ou3s0 https://www.youtube.com/watch?v=Z9ekw2Ou3s0

derriz 7mo ago

If I were investing effort into acquiring knowledge in this domain, I'd skip straight to Polars. Before I made the switch, I had been using Pandas on and off for more than a decade. I'm not sure how representative this is, but most of the people I know who were Pandas users have also made this switch. I initially did it for the performance improvements but the API (according to my subjective opinion) is much more logical and has far fewer surprises compared to Pandas and it would be my default choice for this reason alone at this stage despite my years of Pandas experience.

rithdmc 7mo ago
Thanks, I'll look into this in the future. I don't need the most performant script, but this could change.
- ertgbnm 7mo ago
  It's less about performance and more about ecosystem lockin. It's a bit like imperial vs metric units. Why would you ever chose to learn imperial if you had the option to only ever use metric to begin with?
  - rithdmc 7mo ago
    Because these are silly personal scripts. I'm not going to make sensible architectural decisions on something I run every now and then on my laptop. That's optimising too early.
  - short_sells_poo 7mo ago
    For short scripts and interactive research work, pandas is still much better than polars. Polars works well when you know what you want. When you are still figuring out things step by step, pandas does a lot of heavy lifting for you so you don't have to think about it. E.g. I don't have to think about timeseries alignment, pandas handles that for me implicitly because dataframes can be indexed by timestamps. Polars has timeseries support, but I need to write a paragraph of extra code to deal with it.
  - dec0dedab0de 7mo ago
    That's exactly why I am reluctant to do anything with Polars. They are actively running a company and trying to sell a product. At any point they could be acquired and change the license for new releases. Sure you could fork it, or stay on an older version, but if what they offer isn't compelling enough for you then why take the risk? Pandas on the other hand has been open source for almost two decades, and is supported by many companies. They have a governance board, and an active community. The risk of it going off the rails into corporate nonsense is much lower.