4 ms·

Did the author beat pandas group an aggregate by using standard Python lists?

morelisp 4y ago
The main optimization at that stage seems to be preallocating the weights. I don't know pandas but such a thing would have been possible without dropping any of the linalg libraries I do know how to use. I doubt the author's C++ implementations beat BLAS/LAPACK, but since they're not shown I can only guess. I've done stuff like this before but the tooling is really no fun, somewhere between 2 and 3 I'd just write it all in C++. Changing the interface just to get parallelism out seems not great - give it to the user for free if the array is long enough - but maybe it was more reasonable for the non-trivial real problem.
BiteCode_dev 4y ago
Most likely a missued of Pandas. DF are heavy to create, but calculations on them are fast if you stay in the numpy world and stay vectorized.
- jjoonathan 4y ago
  "It's fast so long as you don't use any of the many parts that aren't fast!" This isn't great.
  - deleted 4y ago
    [deleted]
  - BiteCode_dev 4y ago
    That's true for everything in computing. Don't use a hammer as a screwdriver. I'm not even implying they shouldn't have used pandas for this, I'm suggesting they probably wrote the wrong pandas code for this. Pandas is typically 3 times faster than raw Python, not 10 times slower.
    - jjoonathan 4y ago
      No, I think it is fair to call out mediocrity, even when it tries to pull the "disclaim exactly the set of specific applications it gets called out on" trick. Sure, pandas often beats raw python by a bit, but come on, there's so much mediocrity between the two that I doubt they even had to cheat to find a situation the other way around.
      - BiteCode_dev 4y ago
        I wish for myself to create any project in my life that reach twice this level of mediocrity then.
    - cs137 4y ago
      I used to be a hardcore functional programming weenie, but over time I realized that to do high-performance, systems programming in an FP language means writing a bunch of non-idiomatic code, to the point that it's worth considering C (or C++ for STL only, but not that OOP stuff) instead unless you have a good reason (which you might) for a nonstandard language. The problem isn't Python itself. Python has come a long way from where it started. The problem is people using Python for modules where they actually end up needing, say, manual memory management or heterogeneous high performance (e.g. Monte Carlo algorithms).
  - kzrdude 4y ago
    People create accidentally quadratic code all the time. It's even easier in pandas because the feature set is so huge and finding the right way to do it takes some experience (see stackoverflow for a lot of plain loops over pandas dataframes).
- klyrs 4y ago
  Yeah, I'm a bit suspicious that they made two simultaneous changes: 1. remove pandas 2. externalize WEIGHTS and don't generate it every run Point 2 is likely a huge portion of the runtime.
bornfreddy 4y ago
Yes. Apparently they have used Python lists to beat highly optimized library which builds upon numpy, in C. Yeah right. Note that I'm not saying that their second version of the code wasn't faster, just that this has nothing to do with python vs. pandas.
- snovv_crash 4y ago
  Every time I've used tools that used Pandas things are slow. I totally believe this.
- dfgqeryqe 4y ago
  You would think that, wouldn't you? But every time I've worked on a Python code base I have torn out Pandas and replaced it with simple procedural code, getting at least an order of magnitude. Pandas is spectacularly slow. I don't understand how or why, but it is.
  - julius_deane 4y ago
    the reason is here: > I don't understand how or why

The main optimization at that stage seems to be preallocating the weights. I don't know pandas but such a thing would have been possible without dropping any of the linalg libraries I do know how to use. I doubt the author's C++ implementations beat BLAS/LAPACK, but since they're not shown I can only guess. I've done stuff like this before but the tooling is really no fun, somewhere between 2 and 3 I'd just write it all in C++. Changing the interface just to get parallelism out seems not great - give it to the user for free if the array is long enough - but maybe it was more reasonable for the non-trivial real problem.

BiteCode_dev 4y ago

Most likely a missued of Pandas. DF are heavy to create, but calculations on them are fast if you stay in the numpy world and stay vectorized.

jjoonathan 4y ago
"It's fast so long as you don't use any of the many parts that aren't fast!" This isn't great.
- deleted 4y ago
  [deleted]
- BiteCode_dev 4y ago
  That's true for everything in computing. Don't use a hammer as a screwdriver. I'm not even implying they shouldn't have used pandas for this, I'm suggesting they probably wrote the wrong pandas code for this. Pandas is typically 3 times faster than raw Python, not 10 times slower.
  - jjoonathan 4y ago
    No, I think it is fair to call out mediocrity, even when it tries to pull the "disclaim exactly the set of specific applications it gets called out on" trick. Sure, pandas often beats raw python by a bit, but come on, there's so much mediocrity between the two that I doubt they even had to cheat to find a situation the other way around.
    - BiteCode_dev 4y ago
      I wish for myself to create any project in my life that reach twice this level of mediocrity then.
  - cs137 4y ago
    I used to be a hardcore functional programming weenie, but over time I realized that to do high-performance, systems programming in an FP language means writing a bunch of non-idiomatic code, to the point that it's worth considering C (or C++ for STL only, but not that OOP stuff) instead unless you have a good reason (which you might) for a nonstandard language. The problem isn't Python itself. Python has come a long way from where it started. The problem is people using Python for modules where they actually end up needing, say, manual memory management or heterogeneous high performance (e.g. Monte Carlo algorithms).
- kzrdude 4y ago
  People create accidentally quadratic code all the time. It's even easier in pandas because the feature set is so huge and finding the right way to do it takes some experience (see stackoverflow for a lot of plain loops over pandas dataframes).
klyrs 4y ago
Yeah, I'm a bit suspicious that they made two simultaneous changes: 1. remove pandas 2. externalize WEIGHTS and don't generate it every run Point 2 is likely a huge portion of the runtime.

bornfreddy 4y ago

Yes. Apparently they have used Python lists to beat highly optimized library which builds upon numpy, in C. Yeah right. Note that I'm not saying that their second version of the code wasn't faster, just that this has nothing to do with python vs. pandas.

snovv_crash 4y ago
Every time I've used tools that used Pandas things are slow. I totally believe this.
dfgqeryqe 4y ago
You would think that, wouldn't you? But every time I've worked on a Python code base I have torn out Pandas and replaced it with simple procedural code, getting at least an order of magnitude. Pandas is spectacularly slow. I don't understand how or why, but it is.
- julius_deane 4y ago
  the reason is here: > I don't understand how or why