6 ms·

Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but

Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but its one of those pieces of software that has a very non-intuitive API. Does any one find it difficult to use it or its just me? In particular, I find this answer infuriating [1]. I've come across it so many times. Look, I have a CSV with 200 rows and I need to loop through them in the most intuitive way. Sure, its not optimal but I don't want fast code. I have a mental model of how to modify this dataframe. Let me do it, please. [1] https://stackoverflow.com/a/55557758 https://stackoverflow.com/a/55557758

jzwinck 6y ago
Here are two options which various people might find intuitive: 1. Make a good old numeric for loop like "for ii in range(len(df))" with df.iloc or df.ix, etc. 2. Use df.apply() to create a new DataFrame with your changes. Both of these are mentioned in brief answers to that SO post. But not in the accepted answer. A lot of the answers focus on the most efficient ways to do things, even though the question was very basic and did not imply the data were large.
- smabie 6y ago
  It's not even about performance, writing vectorized code is just superior. The array language model is very powerful and you can literally write a couple lines of code that would take hundreds of lines of normal Python. That being said, Pandas isn't a very good array language. I really like kdb+/q and find it better and more expressive than almost any other language I've used.
qwantim1 6y ago
I just read the GitHub repo mentioned in the article and it looks like gibberish to me. Pandas to me are still big lazy black and white bears eating bamboo, until someone can point me to something more intelligible.
chaps 6y ago
100% my experience whenever I work with it, and I've been working with it on and off for about five years. I get its appeal -- it fits that fuzzy place where a database is too heavy, but vanilla python is cumbersome... but damn, is it tough to work with. With almost no fail, I always seem to just scrap the pandas code in favor of vanilla python, either from usability issues, or from yet another a library's false promise that it works well with dataframes. So many hours lost in using pandas. That said, one of its best features, and probably the only thing I use it for these days is, `pd.read_sql(sqlstr, conn).to_csv(fp)`. This is far less cumbersome than using psycopg2. Edit: for charting these days, similar to OP's visualiztions, I highly recommend vega-lite.
- smabie 6y ago
  Pandas isn't great, but I find it significantly nicer to use than normal Python. Why do you want to write a 100 lines of dumb loopy code when you can probably solve the same problem with a couple lines of Pandas? If you really just want to use loops and stuff (which I would discourage) just use a list or a dict or something.
  - ab111111111 6y ago
    As you indicate, Pandas is effectively another language, distinct from regular Python. E.g. it doesn't really use loops. So I think what irritates a lot of people about pandas is that they think "Cool. I can solve this complex problem in a couple of lines of Python code with pandas." then get irritated when they find that the couple of lines of pandas they need to use is "Incomprehensible pandas gibberish" instead of the familiar Python code they were expecting.
  - deleted 6y ago
    [deleted]
  - chaps 6y ago
    "If you really just want to use loops and stuff (which I would discourage) just use a list or a dict or something." Like my post said, I favor vanilla python over pandas... so yes, I use lists, dicts and somethings. FWIW, though, my workflow pushes everything into postgres, and things that would normally go into pandas are just accessed through SQL through and with helper functions.
dwohnitmok 6y ago
I understand where you're coming from (yes yes there's a theoretically "good" way of doing this, but come on, why can't I just do the simplest thing), but I also sympathize with the spirit of that SO answer (although I agree that its presentation is wanting). Pandas is really a DSL unto itself and is heavily influenced by R, where the same dynamic happens. Programmers coming from a background where procedural control flow constructs are basically second nature bump up against statisticians for whom array-based programming (in the form of overloaded mathematical notation acting on both scalar and vector values) is second nature. R and pandas are both very array-oriented programming languages (the most extreme example of this might be early-era APL) and it's really going against the grain to implement things with explicit iteration. It's kind of like trying to program in Python without using loops or list comprehensions and asking just how to do everything in recursion. You can... but someone is bound to point out that doing everything with recursion (and the concomitant trampolines to prevent stack overflows) is not the Pythonic way. (Also separately @dang, I feel like I'm running into a very minor bug with time stamps, where when I'm composing this reply I get "9 hours ago" for systemvoltage, but in the main thread I get "3 hours ago")
- smabie 6y ago
  I actually don't like Pandas but I think it's pretty obvious that the array language style is an incredibly powerful way to manipulate blocks of multiple dimensional data. People here are complaining about not being able to use for loops, but why would you want to use them in the first place? Like let's say you want to add two vectors together. Looping over each index and adding them and assigning them to a third vector element by element is not only I expressive (and like everything else written in Python, computational inefficient), but it is an conceptually inappropriate solution to the problem. You are operating on n-dimensional arrays, not elements, so your need to write code that expresses that intent. Anyone who thinks that anything besides writing "a + b" to add two matrices together is a good or simple solution is crazy. Operate at a higher conceptual level. Don't use loops. Transform and compose your data, not your datums.
  - edmundsauto 6y ago
    For folks doing this everyday, absolutely. For anyone just learning, it's Yet Another Thing to learn. Sure, the complete programmer will be able to think in better abstractions, but most people fail to reach that level. In part because of the complexity of early stage learning. It's a balance, and I think pandas is on the more complex side of things.
ab111111111 6y ago
Same here. Why re-invent SQL as a weird object system? I've also recently found that using the sqlite3 command line tool increases my productivity when doing data sciency stuff a lot. It's a super fast, super simple way of making sense of CSV data, especially for the selecting an joining operations that are so unintuitive in pandas. Once that's done, I can dump the results into Jupyter or RStudio for transformation and/or visualization. Another personal productivity win I've discovered is using two-line python scripts to write really long and repetitive SQL commands (select count a, b, c... from huge_table where d... and e... and f...; etc.) and then running them in SQLite.
dehrmann 6y ago
I'm in this boat, and I'm decent at SQL and find the Numpy API to be pretty intuitive, so I'm not sure what it is.
philshem 6y ago
It has happened that I’ve searched StackOverflow with Pandas questions and found an answer written by me.
kache_ 6y ago
it's shit
greenie_beans 6y ago
This is reassuring. Anytime I use pandas I’m like wtf? (First encountered it like a year ago)

jzwinck 6y ago

Here are two options which various people might find intuitive: 1. Make a good old numeric for loop like "for ii in range(len(df))" with df.iloc or df.ix, etc. 2. Use df.apply() to create a new DataFrame with your changes. Both of these are mentioned in brief answers to that SO post. But not in the accepted answer. A lot of the answers focus on the most efficient ways to do things, even though the question was very basic and did not imply the data were large.

smabie 6y ago
It's not even about performance, writing vectorized code is just superior. The array language model is very powerful and you can literally write a couple lines of code that would take hundreds of lines of normal Python. That being said, Pandas isn't a very good array language. I really like kdb+/q and find it better and more expressive than almost any other language I've used.

qwantim1 6y ago

I just read the GitHub repo mentioned in the article and it looks like gibberish to me. Pandas to me are still big lazy black and white bears eating bamboo, until someone can point me to something more intelligible.

chaps 6y ago

100% my experience whenever I work with it, and I've been working with it on and off for about five years. I get its appeal -- it fits that fuzzy place where a database is too heavy, but vanilla python is cumbersome... but damn, is it tough to work with. With almost no fail, I always seem to just scrap the pandas code in favor of vanilla python, either from usability issues, or from yet another a library's false promise that it works well with dataframes. So many hours lost in using pandas. That said, one of its best features, and probably the only thing I use it for these days is, `pd.read_sql(sqlstr, conn).to_csv(fp)`. This is far less cumbersome than using psycopg2. Edit: for charting these days, similar to OP's visualiztions, I highly recommend vega-lite.

smabie 6y ago
Pandas isn't great, but I find it significantly nicer to use than normal Python. Why do you want to write a 100 lines of dumb loopy code when you can probably solve the same problem with a couple lines of Pandas? If you really just want to use loops and stuff (which I would discourage) just use a list or a dict or something.
- ab111111111 6y ago
  As you indicate, Pandas is effectively another language, distinct from regular Python. E.g. it doesn't really use loops. So I think what irritates a lot of people about pandas is that they think "Cool. I can solve this complex problem in a couple of lines of Python code with pandas." then get irritated when they find that the couple of lines of pandas they need to use is "Incomprehensible pandas gibberish" instead of the familiar Python code they were expecting.
- deleted 6y ago
  [deleted]
- chaps 6y ago
  "If you really just want to use loops and stuff (which I would discourage) just use a list or a dict or something." Like my post said, I favor vanilla python over pandas... so yes, I use lists, dicts and somethings. FWIW, though, my workflow pushes everything into postgres, and things that would normally go into pandas are just accessed through SQL through and with helper functions.

dwohnitmok 6y ago

I understand where you're coming from (yes yes there's a theoretically "good" way of doing this, but come on, why can't I just do the simplest thing), but I also sympathize with the spirit of that SO answer (although I agree that its presentation is wanting). Pandas is really a DSL unto itself and is heavily influenced by R, where the same dynamic happens. Programmers coming from a background where procedural control flow constructs are basically second nature bump up against statisticians for whom array-based programming (in the form of overloaded mathematical notation acting on both scalar and vector values) is second nature. R and pandas are both very array-oriented programming languages (the most extreme example of this might be early-era APL) and it's really going against the grain to implement things with explicit iteration. It's kind of like trying to program in Python without using loops or list comprehensions and asking just how to do everything in recursion. You can... but someone is bound to point out that doing everything with recursion (and the concomitant trampolines to prevent stack overflows) is not the Pythonic way. (Also separately @dang, I feel like I'm running into a very minor bug with time stamps, where when I'm composing this reply I get "9 hours ago" for systemvoltage, but in the main thread I get "3 hours ago")

smabie 6y ago
I actually don't like Pandas but I think it's pretty obvious that the array language style is an incredibly powerful way to manipulate blocks of multiple dimensional data. People here are complaining about not being able to use for loops, but why would you want to use them in the first place? Like let's say you want to add two vectors together. Looping over each index and adding them and assigning them to a third vector element by element is not only I expressive (and like everything else written in Python, computational inefficient), but it is an conceptually inappropriate solution to the problem. You are operating on n-dimensional arrays, not elements, so your need to write code that expresses that intent. Anyone who thinks that anything besides writing "a + b" to add two matrices together is a good or simple solution is crazy. Operate at a higher conceptual level. Don't use loops. Transform and compose your data, not your datums.
- edmundsauto 6y ago
  For folks doing this everyday, absolutely. For anyone just learning, it's Yet Another Thing to learn. Sure, the complete programmer will be able to think in better abstractions, but most people fail to reach that level. In part because of the complexity of early stage learning. It's a balance, and I think pandas is on the more complex side of things.

ab111111111 6y ago

Same here. Why re-invent SQL as a weird object system? I've also recently found that using the sqlite3 command line tool increases my productivity when doing data sciency stuff a lot. It's a super fast, super simple way of making sense of CSV data, especially for the selecting an joining operations that are so unintuitive in pandas. Once that's done, I can dump the results into Jupyter or RStudio for transformation and/or visualization. Another personal productivity win I've discovered is using two-line python scripts to write really long and repetitive SQL commands (select count a, b, c... from huge_table where d... and e... and f...; etc.) and then running them in SQLite.

dehrmann 6y ago

I'm in this boat, and I'm decent at SQL and find the Numpy API to be pretty intuitive, so I'm not sure what it is.

philshem 6y ago

It has happened that I’ve searched StackOverflow with Pandas questions and found an answer written by me.

kache_ 6y ago

it's shit

greenie_beans 6y ago

This is reassuring. Anytime I use pandas I’m like wtf? (First encountered it like a year ago)