3 ms·

I always see these type of complaints and, when I actually sit down with people to resolve their aversion, it ultimately comes back to they use Pandas incorrect

I always see these type of complaints and, when I actually sit down with people to resolve their aversion, it ultimately comes back to they use Pandas incorrectly or simply are not able to grok documentation. The usual dead giveaway is "Pandas documentation is horrible". Anyone who has used more than one documentation would know that documentation rarely includes every function in the API let alone the argument, examples, and links to other related functions. As this is the top comment, can you (and others) at least post the problems so that we can have an intellectual discussion? Maybe the Pandas devs might take a point or two.

brundolf 4y ago
In my relatively brief experience maintaining a Python web service backed by Pandas: It's basically a DSL constructed out of the dismembered syntactic bones of Python, which breaks every piece of semantics in the host language that it possibly can. I'm sure this is convenient (and maybe even tractable to use) in an interactive notebook, where you can try out and verify behavior in real-time by looking at the output. But in any kind of non-immediate-feedback scenario where you're trying to engineer a production system, it's a hellscape of jumping back and forth to the documentation because even your most fundamental assumptions about the host language's semantics have been thrown out the window. On top of that (and largely because of it), it also resists static typing like crazy. It also has a bunch of APIs where it's really hard to know what will and won't mutate, and others that are just made generally hard to wrap your brain around for the sake of saving a few characters. Finally: some portion of the hate it gets is probably from engineers without a statistics background who aren't familiar with the huge dictionary of jargon and abbreviations its APIs use. This complaint is maybe less valid, since it is a data science library, but it certainly colors emotions and makes it even harder to deal with in a production context for lots of us. There were several times where, once I did some research and dug past the jargon and learned all the background for what it meant, the concept itself wasn't complicated. But Pandas did use the jargon, so I couldn't just understand, I had to go learn stats first and traverse all this needless indirection. It's an API that isn't designed for engineers but engineers often have to deal with it anyway.
- otsaloma 4y ago
  > Finally: some portion of the hate it gets is probably from engineers without a statistics background who aren't familiar with the huge dictionary of jargon and abbreviations its APIs use. Pandas' jargon isn't really even statistics jargon. For example data frames are tables, which are inherently rows and columns. Columns have names, rows can have names too (although that's not really needed). Pandas' documentation has very few mentions of rows and columns at all, instead they use "axes", "labels" and "index" and there's not even a proper explanation in documentation of what those mean. And that "index" is something that the user needs to manage, sometimes "drop", sometimes "reset" without really understanding why. It seems to me two things: (1) really bad choice of naming things and (2) exposing to users some performance-related details that should have been kept under the hood, if they are needed at all.
- hervature 4y ago
  I would put this squarely in the "use Pandas incorrectly" bucket again. As you point out, Pandas is very much geared towards notebooks. Conversely, I would claim your problem is actually one of the benefits of Pandas over something like R and the associated packages. At the very least, you can copy your Pandas code and ship it and collect technical debt. With R, either you make calls to R from Python, which doesn't solve the problem you are having, or you rewrite the code in Python to be usable in your app. This costs time and is the correct way to do things, even with Pandas. I'm sorry you had to handle the technical debt but I hope that was an intentional decision on your project's behalf. As for your other complaints, I would need to see examples because I'm not really sure what you mean. Are you referring to things like "mad" which stands for Mean Absolute Deviation? This I would put into my "grok documentation" bucket because just look up the mad method and it literally gives you the acronym. Would the method "mean_absolute_deviation" really help you? Furthermore, if they used long names like that, statisticians from R (which are the target audience) would have another nothing-burger bullet to use against Pandas.
- deleted 4y ago
  [deleted]

brundolf 4y ago

In my relatively brief experience maintaining a Python web service backed by Pandas: It's basically a DSL constructed out of the dismembered syntactic bones of Python, which breaks every piece of semantics in the host language that it possibly can. I'm sure this is convenient (and maybe even tractable to use) in an interactive notebook, where you can try out and verify behavior in real-time by looking at the output. But in any kind of non-immediate-feedback scenario where you're trying to engineer a production system, it's a hellscape of jumping back and forth to the documentation because even your most fundamental assumptions about the host language's semantics have been thrown out the window. On top of that (and largely because of it), it also resists static typing like crazy. It also has a bunch of APIs where it's really hard to know what will and won't mutate, and others that are just made generally hard to wrap your brain around for the sake of saving a few characters. Finally: some portion of the hate it gets is probably from engineers without a statistics background who aren't familiar with the huge dictionary of jargon and abbreviations its APIs use. This complaint is maybe less valid, since it is a data science library, but it certainly colors emotions and makes it even harder to deal with in a production context for lots of us. There were several times where, once I did some research and dug past the jargon and learned all the background for what it meant, the concept itself wasn't complicated. But Pandas did use the jargon, so I couldn't just understand, I had to go learn stats first and traverse all this needless indirection. It's an API that isn't designed for engineers but engineers often have to deal with it anyway.

otsaloma 4y ago
> Finally: some portion of the hate it gets is probably from engineers without a statistics background who aren't familiar with the huge dictionary of jargon and abbreviations its APIs use. Pandas' jargon isn't really even statistics jargon. For example data frames are tables, which are inherently rows and columns. Columns have names, rows can have names too (although that's not really needed). Pandas' documentation has very few mentions of rows and columns at all, instead they use "axes", "labels" and "index" and there's not even a proper explanation in documentation of what those mean. And that "index" is something that the user needs to manage, sometimes "drop", sometimes "reset" without really understanding why. It seems to me two things: (1) really bad choice of naming things and (2) exposing to users some performance-related details that should have been kept under the hood, if they are needed at all.
hervature 4y ago
I would put this squarely in the "use Pandas incorrectly" bucket again. As you point out, Pandas is very much geared towards notebooks. Conversely, I would claim your problem is actually one of the benefits of Pandas over something like R and the associated packages. At the very least, you can copy your Pandas code and ship it and collect technical debt. With R, either you make calls to R from Python, which doesn't solve the problem you are having, or you rewrite the code in Python to be usable in your app. This costs time and is the correct way to do things, even with Pandas. I'm sorry you had to handle the technical debt but I hope that was an intentional decision on your project's behalf. As for your other complaints, I would need to see examples because I'm not really sure what you mean. Are you referring to things like "mad" which stands for Mean Absolute Deviation? This I would put into my "grok documentation" bucket because just look up the mad method and it literally gives you the acronym. Would the method "mean_absolute_deviation" really help you? Furthermore, if they used long names like that, statisticians from R (which are the target audience) would have another nothing-burger bullet to use against Pandas.
deleted 4y ago
[deleted]