7 ms·

Yeah I can see why 3500 pages are needed to document this inconsistent piece of software. Can never tell whether the method I have in mind is static or not, or

Yeah I can see why 3500 pages are needed to document this inconsistent piece of software. Can never tell whether the method I have in mind is static or not, or how to access elements, (is it loc, iloc, query, [] directly, etc), will the results be a dataframe or a series or straight up numpy array/lists, index issues, etc etc. If you think that these complaints are not valid or stem from a lack of experience, I urge you to use R first.

bradrn 5y ago
Huh, so it wasn’t just me who was confused when I tried to use pandas. I found pretty much anything non-trivial to be difficult at best and impossible at worst. (IIRC I needed to sum rows grouped by time period or something like that.) Then again, I haven’t used pandas much at all. (I’ve heard tell that R is worse, but I wouldn’t have any opinion on that since I’ve never used the language.)
- dwater 5y ago
  Yeah, I'm not sure I understand the "I urge you to use R first" comment in the grandparent post, because I've used both and R's tidyverse was at least as difficult to approach as pandas for me. And what's the alternative, Excel? So it stands to reason there must be a common explanation whey they are all so obtuse, which I would venture is something like: they solve a set of problems so broad and complex that the tools themselves must also be massively broad and complex, and so the only way to avoid API inconsistencies is to have the system designed whole-cloth from the very beginning, but doing so would make the development process so huge that it would take a decade before you had something ready for users and so it would never get off the ground. The tools that succeed in reality are the ones that solve a common subset of problems, and then grow organically as they are extended to solve other subsets, and the varying subsets have logical inconsistencies and conflicts, thus the state of tools available today having APIs with inconsistencies and conflicts. That said, pandas is really good at solving problems with tabular data, and even though it's not easy to do complex work with I still like and appreciate it very much.
  - nn3 5y ago
    A lot of learning curve come from them essentially being array programming languages (like APL). They are powerful, but just very different if you aren't used to them before. The alternative really is to write simple imperative style scripts with for loops in your favorite scripting language. Script will be much longer and likely more prone to errors, but the learning curve for simple tasks is just a lot less steep. Eventually learning the array oriented approach is worth it though. It's the usual trade off in how much time is it worth learning a new tool vs kludging some old tool that doesn't quite fit for the task.
  - lf-non 5y ago
    > And what's the alternative, Excel? Take this with a grain of salt from someone who needs data manipulation occasionally every now and then (as opposed to being a full time number-cruncher, data-scientist, statistician etc.), using krangl[1] for Kotlin has been a great experience. I was drawn to this library because I use Kotlin in my dayjob for backend development, but I love how well Kotlin's succinct syntax & features like extension functions lends itself to data transformation & ETL kind of use cases. Also it doesn't hurt that JVM has a plethora of libraries available for things like DB access, plotting, etc. I am sure that Pandas has many features I am unaware of, and for a lot of people the high-ish startup time can be a deterrant, but for most of my day to day data munging the combination of jbang, krangl & kravis has been a pretty good fit. [1] https://github.com/holgerbrandl/krangl https://github.com/holgerbrandl/krangl
  - mejutoco 5y ago
    I have used pandas a lot and by now I am used to its quirks, but I still find R to be much more consistent. Just another data point.
  - hermitcrab 5y ago
    >And what's the alternative, Excel? Lots of GUI based alternatives for data munging such as: Easy Data Transform, Alteryx, Knime. Whether they are better than text based tools, such as R or Python+Pandas, depends a lot on your use case and experience.
mjburgess 5y ago
Python wasn't designed with vectorised data structures in mind, `R` was -- and that's the origin of a vast number of issues. Data Science happened to python largely against its will (ie., that of GvR) -- and language-level support simply doesn't exist.
- wodenokoto 5y ago
  Pandas is an incredible piece of software, but it is also a confusing, hard to use and inconsistent api. It can be both, but it doesn’t have to.
  - mjburgess 5y ago
    Data analysis isn't object-oriented nor imperative. Transformations are functions on datasets, not a sequence of stateful actions. Python is an imperative, object-oriented language largely concerned with side-effecting programs (web severs, os automation, etc.). It is about as hostile as it is possible to be, from "ideas" pov, to data science. However those ideas are just a syntax skin over something much more flexible: pointers and dictionaries (/hashmaps). These are like the "Python VM Language" on which "Data Science" sits. Data Science libraries essentially use the "actual" semantics of python and ignore the "official" syntax, as that's the only way of providing an API that makes any sense. (Ie., isnt just reams of for-loops with 5 sequential obj.method() calls). I don't believe there is a "non-confusing" way of ignore python's syntax, *within python*; and that's just what is required.
    - lostmsu 5y ago
      What a weird notion. Why do you think building transformations is not a good use for an (optionally) imperative OOP language?
      - aaron-santos 5y ago
        When working with data, I like to avoid stateful data mutations at all cost. OOP's focus is mutating state. These are at odds. The identity semantics I hold for data are not invariant under transformation.
_Wintermute 5y ago
If anyone thinks pandas has a lot of inconsistencies and unexpected behaviour, you're in for a wild ride with R.
- CornCobs 5y ago
  I beg to differ. If you are talking about dplyr, it has a much smaller and consistent API. And I learned python first. It is certainly interesting though, how there are so many people firmly in opposite camps. I really cannot imagine how pandas could be considered well put together; I'm always searching stackoverflow for data transformations I assume should be common and simple. In dplyr, on the other hand, each function does one and only one thing (the touted Unix principle) and you just pass the data along via the pipe %>% operator
  - _Wintermute 5y ago
    dplyr is nice, but the fact there is an entire company writing packages to get around R's inconsistent standard library might be a bit of a hint. Also, as soon as you stray away from dataframes R is completely lost.
BeetleB 5y ago
I recently inherited an R codebase at work. It was not fun understanding it, and trying to hunt down docs to explain some of the functions was a much worse experience than it is with Pandas[1]. So I did the sane thing: I figured it all out and rewrote it in Pandas. [1] I have never failed to find an answer for Pandas on SO for even the most convoluted manipulations I'm attempting. The first hit I look at usually has my answer.
mejutoco 5y ago
Cannot agree more. Even using R just to read csvs, transform the data, and output another csv, can be an improvement, and it lets you still use python for other things.
mands 5y ago
How is Julia in this regard?

bradrn 5y ago

Huh, so it wasn’t just me who was confused when I tried to use pandas. I found pretty much anything non-trivial to be difficult at best and impossible at worst. (IIRC I needed to sum rows grouped by time period or something like that.) Then again, I haven’t used pandas much at all. (I’ve heard tell that R is worse, but I wouldn’t have any opinion on that since I’ve never used the language.)

dwater 5y ago
Yeah, I'm not sure I understand the "I urge you to use R first" comment in the grandparent post, because I've used both and R's tidyverse was at least as difficult to approach as pandas for me. And what's the alternative, Excel? So it stands to reason there must be a common explanation whey they are all so obtuse, which I would venture is something like: they solve a set of problems so broad and complex that the tools themselves must also be massively broad and complex, and so the only way to avoid API inconsistencies is to have the system designed whole-cloth from the very beginning, but doing so would make the development process so huge that it would take a decade before you had something ready for users and so it would never get off the ground. The tools that succeed in reality are the ones that solve a common subset of problems, and then grow organically as they are extended to solve other subsets, and the varying subsets have logical inconsistencies and conflicts, thus the state of tools available today having APIs with inconsistencies and conflicts. That said, pandas is really good at solving problems with tabular data, and even though it's not easy to do complex work with I still like and appreciate it very much.
- nn3 5y ago
  A lot of learning curve come from them essentially being array programming languages (like APL). They are powerful, but just very different if you aren't used to them before. The alternative really is to write simple imperative style scripts with for loops in your favorite scripting language. Script will be much longer and likely more prone to errors, but the learning curve for simple tasks is just a lot less steep. Eventually learning the array oriented approach is worth it though. It's the usual trade off in how much time is it worth learning a new tool vs kludging some old tool that doesn't quite fit for the task.
- lf-non 5y ago
  > And what's the alternative, Excel? Take this with a grain of salt from someone who needs data manipulation occasionally every now and then (as opposed to being a full time number-cruncher, data-scientist, statistician etc.), using krangl[1] for Kotlin has been a great experience. I was drawn to this library because I use Kotlin in my dayjob for backend development, but I love how well Kotlin's succinct syntax & features like extension functions lends itself to data transformation & ETL kind of use cases. Also it doesn't hurt that JVM has a plethora of libraries available for things like DB access, plotting, etc. I am sure that Pandas has many features I am unaware of, and for a lot of people the high-ish startup time can be a deterrant, but for most of my day to day data munging the combination of jbang, krangl & kravis has been a pretty good fit. [1] https://github.com/holgerbrandl/krangl https://github.com/holgerbrandl/krangl
- mejutoco 5y ago
  I have used pandas a lot and by now I am used to its quirks, but I still find R to be much more consistent. Just another data point.
- hermitcrab 5y ago
  >And what's the alternative, Excel? Lots of GUI based alternatives for data munging such as: Easy Data Transform, Alteryx, Knime. Whether they are better than text based tools, such as R or Python+Pandas, depends a lot on your use case and experience.

mjburgess 5y ago

Python wasn't designed with vectorised data structures in mind, `R` was -- and that's the origin of a vast number of issues. Data Science happened to python largely against its will (ie., that of GvR) -- and language-level support simply doesn't exist.

wodenokoto 5y ago
Pandas is an incredible piece of software, but it is also a confusing, hard to use and inconsistent api. It can be both, but it doesn’t have to.
- mjburgess 5y ago
  Data analysis isn't object-oriented nor imperative. Transformations are functions on datasets, not a sequence of stateful actions. Python is an imperative, object-oriented language largely concerned with side-effecting programs (web severs, os automation, etc.). It is about as hostile as it is possible to be, from "ideas" pov, to data science. However those ideas are just a syntax skin over something much more flexible: pointers and dictionaries (/hashmaps). These are like the "Python VM Language" on which "Data Science" sits. Data Science libraries essentially use the "actual" semantics of python and ignore the "official" syntax, as that's the only way of providing an API that makes any sense. (Ie., isnt just reams of for-loops with 5 sequential obj.method() calls). I don't believe there is a "non-confusing" way of ignore python's syntax, *within python*; and that's just what is required.
  - lostmsu 5y ago
    What a weird notion. Why do you think building transformations is not a good use for an (optionally) imperative OOP language?
    - aaron-santos 5y ago
      When working with data, I like to avoid stateful data mutations at all cost. OOP's focus is mutating state. These are at odds. The identity semantics I hold for data are not invariant under transformation.

_Wintermute 5y ago

If anyone thinks pandas has a lot of inconsistencies and unexpected behaviour, you're in for a wild ride with R.

CornCobs 5y ago
I beg to differ. If you are talking about dplyr, it has a much smaller and consistent API. And I learned python first. It is certainly interesting though, how there are so many people firmly in opposite camps. I really cannot imagine how pandas could be considered well put together; I'm always searching stackoverflow for data transformations I assume should be common and simple. In dplyr, on the other hand, each function does one and only one thing (the touted Unix principle) and you just pass the data along via the pipe %>% operator
- _Wintermute 5y ago
  dplyr is nice, but the fact there is an entire company writing packages to get around R's inconsistent standard library might be a bit of a hint. Also, as soon as you stray away from dataframes R is completely lost.

BeetleB 5y ago

I recently inherited an R codebase at work. It was not fun understanding it, and trying to hunt down docs to explain some of the functions was a much worse experience than it is with Pandas[1]. So I did the sane thing: I figured it all out and rewrote it in Pandas. [1] I have never failed to find an answer for Pandas on SO for even the most convoluted manipulations I'm attempting. The first hit I look at usually has my answer.

mejutoco 5y ago

Cannot agree more. Even using R just to read csvs, transform the data, and output another csv, can be an improvement, and it lets you still use python for other things.

mands 5y ago

How is Julia in this regard?