3 ms·

Pandas is an absolute horrorshow: poor performance, inconsistent API, terrible implicit behavior leading to footguns. And everyone uses it because it's what yo

Pandas is an absolute horrorshow: poor performance, inconsistent API, terrible implicit behavior leading to footguns. And everyone uses it because it's what you do when your boss tells you "we're transforming the analytics team, you're all to become data scientists because everyone has data scientists now". You just grab whatever had the biggest mindshare on SO and in random yt tutorials. Can't blame them. But hooo boy does pd get on my nerves. Care to share what propriety stuff you were using?

singhrac 5y ago
I can't really share in any detail, I think, but the best part was that "Series" were immutable and had sorted keys (indexes). Essentially they were (math) functions, so "indexes" had unique elements. All the important bits had fast numpy/Cython implementations, but the semantics were good because of unique keys. Honestly I still feel like I'm missing some sort of larger story about the semantics of Pandas (like the "functions" explanation above), so if anyone knows of anything that made Pandas click, please let me know.
- radus 5y ago
  This addresses some of my misgivings with pandas: https://tomaugspurger.github.io/method-chaining.html https://tomaugspurger.github.io/method-chaining.html
  - singhrac 5y ago
    I had time to read through this, thank you for linking it to me. I like method chaining, and appreciate the terseness of the syntax. I don't think it addresses my main complaint (and other's complaints, below) about Pandas. I'll write a lengthy comment here in the hopes that a Pandas/Polars maintainer sees it and gets some inspiration. One big issue I see with Pandas is that it assumes that I want multidimensional keys represented via axes labels on some sort of grid (or hypercube). This is evident from the __getitem__ interface, which is frankly confusing. In reality, selection is the most important detail, and managing columns is just a nice-to-have. Pandas's MVP is the Series object, not DataFrame. In article #1 in your link, "SettingWithCopy" is emblematic of the issue that happens when you allow mutability, so mutability delenda est. This is why FridgeSeal is confused below. One day I'll write a blog post about ways Pandas could be better, but in the meantime I will probably monkey patch a lot of the nonsense out.

singhrac 5y ago

I can't really share in any detail, I think, but the best part was that "Series" were immutable and had sorted keys (indexes). Essentially they were (math) functions, so "indexes" had unique elements. All the important bits had fast numpy/Cython implementations, but the semantics were good because of unique keys. Honestly I still feel like I'm missing some sort of larger story about the semantics of Pandas (like the "functions" explanation above), so if anyone knows of anything that made Pandas click, please let me know.

radus 5y ago
This addresses some of my misgivings with pandas: https://tomaugspurger.github.io/method-chaining.html https://tomaugspurger.github.io/method-chaining.html
- singhrac 5y ago
  I had time to read through this, thank you for linking it to me. I like method chaining, and appreciate the terseness of the syntax. I don't think it addresses my main complaint (and other's complaints, below) about Pandas. I'll write a lengthy comment here in the hopes that a Pandas/Polars maintainer sees it and gets some inspiration. One big issue I see with Pandas is that it assumes that I want multidimensional keys represented via axes labels on some sort of grid (or hypercube). This is evident from the __getitem__ interface, which is frankly confusing. In reality, selection is the most important detail, and managing columns is just a nice-to-have. Pandas's MVP is the Series object, not DataFrame. In article #1 in your link, "SettingWithCopy" is emblematic of the issue that happens when you allow mutability, so mutability delenda est. This is why FridgeSeal is confused below. One day I'll write a blog post about ways Pandas could be better, but in the meantime I will probably monkey patch a lot of the nonsense out.