8 ms·

This is mildly off topic, but I am very unhappy with Pandas. Every single API feels bolted on without any consideration of composability or ergonomics. After sp

This is mildly off topic, but I am very unhappy with Pandas. Every single API feels bolted on without any consideration of composability or ergonomics. After spending 4 years with a much better proprietary library I cannot deal with arbitrary functions I have to learn like “value_counts” or whatever the output of a “groupby” is.

isoprophlex 4y ago
Pandas is an absolute horrorshow: poor performance, inconsistent API, terrible implicit behavior leading to footguns. And everyone uses it because it's what you do when your boss tells you "we're transforming the analytics team, you're all to become data scientists because everyone has data scientists now". You just grab whatever had the biggest mindshare on SO and in random yt tutorials. Can't blame them. But hooo boy does pd get on my nerves. Care to share what propriety stuff you were using?
- singhrac 4y ago
  I can't really share in any detail, I think, but the best part was that "Series" were immutable and had sorted keys (indexes). Essentially they were (math) functions, so "indexes" had unique elements. All the important bits had fast numpy/Cython implementations, but the semantics were good because of unique keys. Honestly I still feel like I'm missing some sort of larger story about the semantics of Pandas (like the "functions" explanation above), so if anyone knows of anything that made Pandas click, please let me know.
  - radus 4y ago
    This addresses some of my misgivings with pandas: https://tomaugspurger.github.io/method-chaining.html https://tomaugspurger.github.io/method-chaining.html
    - singhrac 4y ago
      I had time to read through this, thank you for linking it to me. I like method chaining, and appreciate the terseness of the syntax. I don't think it addresses my main complaint (and other's complaints, below) about Pandas. I'll write a lengthy comment here in the hopes that a Pandas/Polars maintainer sees it and gets some inspiration. One big issue I see with Pandas is that it assumes that I want multidimensional keys represented via axes labels on some sort of grid (or hypercube). This is evident from the __getitem__ interface, which is frankly confusing. In reality, selection is the most important detail, and managing columns is just a nice-to-have. Pandas's MVP is the Series object, not DataFrame. In article #1 in your link, "SettingWithCopy" is emblematic of the issue that happens when you allow mutability, so mutability delenda est. This is why FridgeSeal is confused below. One day I'll write a blog post about ways Pandas could be better, but in the meantime I will probably monkey patch a lot of the nonsense out.
shankr 4y ago
yeah even after working for years with pandas, I never feel very confident writing it. I always have to look-up even simpler stuff.
- FridgeSeal 4y ago
  And the view-vs-copy shenanigans - a bunch of the API's will print warnings about using the wrong one, and direct you to some docs, but don't actually tell you whether you've just viewed or copied, and the docs don't make it particularly clear which API does which either.
rdedev 4y ago
My go-to these days is Polars. You get good performance since it uses arrow in the back. Coupled that with built-in lazy evaluation and it's API design it's pretty good for me. There are some caveats you need to be aware though. It doesn't always work as a drop in replacement for pandas
- bsg75 4y ago
  Are you referring to https://www.pola.rs https://www.pola.rs ?
  - rdedev 4y ago
    Yup. My bad. Looks like autocorrect screwed me
- akdor1154 4y ago
  Yup, Polars is awesome for both its performance and its stingy but very well designed API.
- silveraxe93 4y ago
  I tried Polars, but feel like it's not ready for production yet. It's fast! The API is nice. But the documentation is not great. While using pandas I can just open the docs and find all functions and examples of how to use, etc. Polars felt really bare. It's something I'd like to use in personal projects. But other data scientists would not accept using polars as it is today.
  - ritchie46 4y ago
    Author of polars here. What would you say the docs is lacking? Quite curious how we should fill the gap. I understand the gap is large due to 12 years of being the standard for python. There is ton of materials for pandas. We hope to have such a clear API that once the expression language clicks, it is less needed to have the docs open constantly as it should feel natural.
    - 4y ago

isoprophlex 4y ago

Pandas is an absolute horrorshow: poor performance, inconsistent API, terrible implicit behavior leading to footguns. And everyone uses it because it's what you do when your boss tells you "we're transforming the analytics team, you're all to become data scientists because everyone has data scientists now". You just grab whatever had the biggest mindshare on SO and in random yt tutorials. Can't blame them. But hooo boy does pd get on my nerves. Care to share what propriety stuff you were using?

singhrac 4y ago
I can't really share in any detail, I think, but the best part was that "Series" were immutable and had sorted keys (indexes). Essentially they were (math) functions, so "indexes" had unique elements. All the important bits had fast numpy/Cython implementations, but the semantics were good because of unique keys. Honestly I still feel like I'm missing some sort of larger story about the semantics of Pandas (like the "functions" explanation above), so if anyone knows of anything that made Pandas click, please let me know.
- radus 4y ago
  This addresses some of my misgivings with pandas: https://tomaugspurger.github.io/method-chaining.html https://tomaugspurger.github.io/method-chaining.html
  - singhrac 4y ago
    I had time to read through this, thank you for linking it to me. I like method chaining, and appreciate the terseness of the syntax. I don't think it addresses my main complaint (and other's complaints, below) about Pandas. I'll write a lengthy comment here in the hopes that a Pandas/Polars maintainer sees it and gets some inspiration. One big issue I see with Pandas is that it assumes that I want multidimensional keys represented via axes labels on some sort of grid (or hypercube). This is evident from the __getitem__ interface, which is frankly confusing. In reality, selection is the most important detail, and managing columns is just a nice-to-have. Pandas's MVP is the Series object, not DataFrame. In article #1 in your link, "SettingWithCopy" is emblematic of the issue that happens when you allow mutability, so mutability delenda est. This is why FridgeSeal is confused below. One day I'll write a blog post about ways Pandas could be better, but in the meantime I will probably monkey patch a lot of the nonsense out.

shankr 4y ago

yeah even after working for years with pandas, I never feel very confident writing it. I always have to look-up even simpler stuff.

FridgeSeal 4y ago
And the view-vs-copy shenanigans - a bunch of the API's will print warnings about using the wrong one, and direct you to some docs, but don't actually tell you whether you've just viewed or copied, and the docs don't make it particularly clear which API does which either.

rdedev 4y ago

My go-to these days is Polars. You get good performance since it uses arrow in the back. Coupled that with built-in lazy evaluation and it's API design it's pretty good for me. There are some caveats you need to be aware though. It doesn't always work as a drop in replacement for pandas

bsg75 4y ago
Are you referring to https://www.pola.rs https://www.pola.rs ?
- rdedev 4y ago
  Yup. My bad. Looks like autocorrect screwed me
akdor1154 4y ago
Yup, Polars is awesome for both its performance and its stingy but very well designed API.
silveraxe93 4y ago
I tried Polars, but feel like it's not ready for production yet. It's fast! The API is nice. But the documentation is not great. While using pandas I can just open the docs and find all functions and examples of how to use, etc. Polars felt really bare. It's something I'd like to use in personal projects. But other data scientists would not accept using polars as it is today.
- ritchie46 4y ago
  Author of polars here. What would you say the docs is lacking? Quite curious how we should fill the gap. I understand the gap is large due to 12 years of being the standard for python. There is ton of materials for pandas. We hope to have such a clear API that once the expression language clicks, it is less needed to have the docs open constantly as it should feel natural.
  - 4y ago