3 ms·

I have had much fun reading this thread of comments today :) This comparison was never about concrete (optimal) implementations but about comparing exactly the

I have had much fun reading this thread of comments today :) This comparison was never about concrete (optimal) implementations but about comparing exactly the same implementations, doing exactly the same operations. Such approach would be the same for comparing performance of any languages, like C and Java. This is not "the best algo" race. So, after a day of discussion, the result is the following: regardless if the implementation is quadratic or linear, the relative speed of all 3 technologies is (roughly) the same. I could not agree more as this is primary school math: dividing by the same denominator still means that proportions are the same :) I plan to update my blog post, based on many interesting comments you have all provided. I definetely think this analysis should be more complex. Thank you for your insights. I have many great pieces of code for inspiration, but I am struggling with providing exactly the same implementations. So it may cost me some time to provide all 3 optimized implementations that are exactly relevant to each other.

eesmith 9y ago
"the relative speed of all 3 technologies is (roughly) the same" I'll be a bit pedantic. The original essay says "pure Python", which is not a technology but a language. There are several implementations of Python, which I'll argue is the "technology". The PyPy implementation of the pure Python algorithm is 10x faster than the CPython implementation. According to the original essay, that's close to the CPython/NumPy performance, and faster than the CPython/Pandas version. It's tempting to omit PyPy because "nobody uses it in data science". That's a self-fulfilling argument. PyPy is doing amazing work to support both NumPy and Pandas, but it's limited by funding. Why aren't data science people pumping money into PyPy? It would give a huge performance boost even for naive, throw-it-together algorithms. If you don't know why PyPy can do for you, how do you learn if/when you should use it?
- lukasz_km 9y ago
  I have tried PyPy once and it gives huge speed up. I am not sure how it works with Pandas/Numpy. I have also tried Numba JIT. Eeasy to use, but in my experince it saves just up to 10% of time. There is also Cython (https://pandas.pydata.org/pandas-docs/stable/enhancingperf.html https://pandas.pydata.org/pandas-docs/stable/enhancingperf.h...) and it definitely works with Pandas. Have not tried yet.
  - eesmith 9y ago
    If you have a pure Python implementation, then why does it matter how well PyPy works with Pandas and NumPy? My point is a minor one. "Python" isn't a technology. "CPython" and "PyPy" are technologies. Or you can try MicroPython, Iron Python, Jython, and others. In any case, you'll see the best idiomatic Pandas code is reported to be 500x faster than the original, while the best idiomatic Python is reported to be 950x faster. The proportions are not the same, so it's not like "dividing by the same denominator". The rank order is unchanged, though that's a weaker observation than what you wrote.
Radim 9y ago
Alright man, good luck. I know HN can be brutal. But this was really not your best piece (technically; it may have met its marketing / traffic goals).

eesmith 9y ago

"the relative speed of all 3 technologies is (roughly) the same" I'll be a bit pedantic. The original essay says "pure Python", which is not a technology but a language. There are several implementations of Python, which I'll argue is the "technology". The PyPy implementation of the pure Python algorithm is 10x faster than the CPython implementation. According to the original essay, that's close to the CPython/NumPy performance, and faster than the CPython/Pandas version. It's tempting to omit PyPy because "nobody uses it in data science". That's a self-fulfilling argument. PyPy is doing amazing work to support both NumPy and Pandas, but it's limited by funding. Why aren't data science people pumping money into PyPy? It would give a huge performance boost even for naive, throw-it-together algorithms. If you don't know why PyPy can do for you, how do you learn if/when you should use it?

lukasz_km 9y ago
I have tried PyPy once and it gives huge speed up. I am not sure how it works with Pandas/Numpy. I have also tried Numba JIT. Eeasy to use, but in my experince it saves just up to 10% of time. There is also Cython (https://pandas.pydata.org/pandas-docs/stable/enhancingperf.html https://pandas.pydata.org/pandas-docs/stable/enhancingperf.h...) and it definitely works with Pandas. Have not tried yet.
- eesmith 9y ago
  If you have a pure Python implementation, then why does it matter how well PyPy works with Pandas and NumPy? My point is a minor one. "Python" isn't a technology. "CPython" and "PyPy" are technologies. Or you can try MicroPython, Iron Python, Jython, and others. In any case, you'll see the best idiomatic Pandas code is reported to be 500x faster than the original, while the best idiomatic Python is reported to be 950x faster. The proportions are not the same, so it's not like "dividing by the same denominator". The rank order is unchanged, though that's a weaker observation than what you wrote.

Radim 9y ago

Alright man, good luck. I know HN can be brutal. But this was really not your best piece (technically; it may have met its marketing / traffic goals).