6 ms·

I understand where you're coming from (yes yes there's a theoretically "good" way of doing this, but come on, why can't I just do the simplest thing), but I als

I understand where you're coming from (yes yes there's a theoretically "good" way of doing this, but come on, why can't I just do the simplest thing), but I also sympathize with the spirit of that SO answer (although I agree that its presentation is wanting). Pandas is really a DSL unto itself and is heavily influenced by R, where the same dynamic happens. Programmers coming from a background where procedural control flow constructs are basically second nature bump up against statisticians for whom array-based programming (in the form of overloaded mathematical notation acting on both scalar and vector values) is second nature. R and pandas are both very array-oriented programming languages (the most extreme example of this might be early-era APL) and it's really going against the grain to implement things with explicit iteration. It's kind of like trying to program in Python without using loops or list comprehensions and asking just how to do everything in recursion. You can... but someone is bound to point out that doing everything with recursion (and the concomitant trampolines to prevent stack overflows) is not the Pythonic way. (Also separately @dang, I feel like I'm running into a very minor bug with time stamps, where when I'm composing this reply I get "9 hours ago" for systemvoltage, but in the main thread I get "3 hours ago")

smabie 6y ago
I actually don't like Pandas but I think it's pretty obvious that the array language style is an incredibly powerful way to manipulate blocks of multiple dimensional data. People here are complaining about not being able to use for loops, but why would you want to use them in the first place? Like let's say you want to add two vectors together. Looping over each index and adding them and assigning them to a third vector element by element is not only I expressive (and like everything else written in Python, computational inefficient), but it is an conceptually inappropriate solution to the problem. You are operating on n-dimensional arrays, not elements, so your need to write code that expresses that intent. Anyone who thinks that anything besides writing "a + b" to add two matrices together is a good or simple solution is crazy. Operate at a higher conceptual level. Don't use loops. Transform and compose your data, not your datums.
- edmundsauto 6y ago
  For folks doing this everyday, absolutely. For anyone just learning, it's Yet Another Thing to learn. Sure, the complete programmer will be able to think in better abstractions, but most people fail to reach that level. In part because of the complexity of early stage learning. It's a balance, and I think pandas is on the more complex side of things.
  - glenngillen 6y ago
    If you don’t want the conceptual model and features pandas provides, is there a reason you don’t just use regular Python without pandas?
    - edmundsauto 6y ago
      Pandas has really great ingesting / exporting helpers, is easy to chart, and lots of tutorials reach for it first. Those might be more excuses than reasons, but it’s my experience.
- kzrdude 6y ago
  It's a conceptual spreadsheet or table. Why wouldn't a read-write (element values) row iterator always be available? SQL tables can do it and numpy.ndarrays can do it.
- Bostonian 6y ago
  Fortran has had array operations since the 1990 standard, but I believe that programmers who are familiar with that functionality still often use loops involving array elements. I do. Not all calculations conveniently map to array operations, and sometimes you can exit the loop if a condition is met. It's a benefit of a compiled, statically typed language to use loops without suffering a speed penalty.
systemvoltage 6y ago
SQL is more natural to me for this sort of declarative DSL. Is there a Pandas-like package that can use in-memory tables and accept SQL queries? About `pd.read_sql()`. That's totally awesome. I realized that Pandas is a very useful tool for many thousands of developers, I only have a problem with its interface. Obviously, I've been using it for many years for a reason!
- carlineng 6y ago
  DuckDB is a really promising project for just that: https://duckdb.org/ https://duckdb.org/
  - chaps 6y ago
    That's freaking cool, thanks for sharing.
trumpeta 6y ago
You’re making a great point there about procedural mindset being applied to array programming. But the thing is, I feel like array based programming should lend itself naturally to functional approaches. And Pandas does do this to an extent. My problem is that this is super inconsistent. Some things are done as a method call on an object, others by passing the object to a pandas function and others yet by passing a function to a method on an object. This is the major source of frustration for me. Maybe there is some logic to that, but I haven’t found it yet and I think that is a sign of bad API design. Its like PHP to me. All nice and documented but useless without Googling everything

smabie 6y ago

I actually don't like Pandas but I think it's pretty obvious that the array language style is an incredibly powerful way to manipulate blocks of multiple dimensional data. People here are complaining about not being able to use for loops, but why would you want to use them in the first place? Like let's say you want to add two vectors together. Looping over each index and adding them and assigning them to a third vector element by element is not only I expressive (and like everything else written in Python, computational inefficient), but it is an conceptually inappropriate solution to the problem. You are operating on n-dimensional arrays, not elements, so your need to write code that expresses that intent. Anyone who thinks that anything besides writing "a + b" to add two matrices together is a good or simple solution is crazy. Operate at a higher conceptual level. Don't use loops. Transform and compose your data, not your datums.

edmundsauto 6y ago
For folks doing this everyday, absolutely. For anyone just learning, it's Yet Another Thing to learn. Sure, the complete programmer will be able to think in better abstractions, but most people fail to reach that level. In part because of the complexity of early stage learning. It's a balance, and I think pandas is on the more complex side of things.
- glenngillen 6y ago
  If you don’t want the conceptual model and features pandas provides, is there a reason you don’t just use regular Python without pandas?
  - edmundsauto 6y ago
    Pandas has really great ingesting / exporting helpers, is easy to chart, and lots of tutorials reach for it first. Those might be more excuses than reasons, but it’s my experience.
kzrdude 6y ago
It's a conceptual spreadsheet or table. Why wouldn't a read-write (element values) row iterator always be available? SQL tables can do it and numpy.ndarrays can do it.
Bostonian 6y ago
Fortran has had array operations since the 1990 standard, but I believe that programmers who are familiar with that functionality still often use loops involving array elements. I do. Not all calculations conveniently map to array operations, and sometimes you can exit the loop if a condition is met. It's a benefit of a compiled, statically typed language to use loops without suffering a speed penalty.

systemvoltage 6y ago

SQL is more natural to me for this sort of declarative DSL. Is there a Pandas-like package that can use in-memory tables and accept SQL queries? About `pd.read_sql()`. That's totally awesome. I realized that Pandas is a very useful tool for many thousands of developers, I only have a problem with its interface. Obviously, I've been using it for many years for a reason!

carlineng 6y ago
DuckDB is a really promising project for just that: https://duckdb.org/ https://duckdb.org/
- chaps 6y ago
  That's freaking cool, thanks for sharing.

trumpeta 6y ago

You’re making a great point there about procedural mindset being applied to array programming. But the thing is, I feel like array based programming should lend itself naturally to functional approaches. And Pandas does do this to an extent. My problem is that this is super inconsistent. Some things are done as a method call on an object, others by passing the object to a pandas function and others yet by passing a function to a method on an object. This is the major source of frustration for me. Maybe there is some logic to that, but I haven’t found it yet and I think that is a sign of bad API design. Its like PHP to me. All nice and documented but useless without Googling everything