5 ms·

I recently discovered that pypy3 can run all my day to day Python code. It has some issues with slightly different behavior from cpython when using threads but

by hyperion2010 9y ago

I recently discovered that pypy3 can run all my day to day Python code. It has some issues with slightly different behavior from cpython when using threads but other than that I see a 4x speedup on most of my slowest pure python workloads (parsing large rdf files and reserializing them after computing a total order on all their nodes). Huge win for productivity.

civility 9y ago
> I see a 4x speedup on most of my slowest pure python workloads Heh, only 25..250 X to go. We did a direct line for line translation of some numerically intensive code from Python to C++ and saw a literal 1000X speedup. On other projects, it's been more like 100X slower. That says two things: first Python can be really slow, second, for some programs, Python doesn't really save on lines of code over modern C++. I've been very impressed with PyPy however. In testing, it can sometimes sneak up to less than a factor of 2 slower than C. However, the bummer comes when it doesn't hit that mark and you have no idea how to trick the JIT to do better. If it works, great. If it doesn't, you don't have much insight into why. Finally, I've always been able to get Cython to parity with C++. However, when I'm done, I wonder what I gained. The C++ isn't that much more complicated than adequately type annotated Cython.
- joshuamorton 9y ago
  While I'm more of a pythonist than a C-ist, hearing "1000x speedup" and "line for line" to me implies that you aren't writing idiomatic python. Idiomatic python is (often) faster than not, and (often) more difficult to translate to lower level languages. As a simple example, list-comprehensions are faster than loops, and can't be line for line translated into C++.
  - pjmlp 9y ago
    Depends, I bet they can be easily translated to some LINQ like implementation in C++17. List comprehensions are just syntax sugar for map/filter/fold.
    - joshuamorton 9y ago
      A jaunt through SO didn't give anything reasonable. This is what SO had to offer: https://stackoverflow.com/questions/36339533/how-to-generate-vector-like-list-comprehension/36340883 https://stackoverflow.com/questions/36339533/how-to-generate..., and a quick check of new C++17 features didn't show any that would obviously improve on that. There's nothing implicitly stopping you from doing some macro magic to implement it, but its not there naturally.
      - pjmlp 9y ago
        SO is not the only source of truth. #include "cpplinq.hpp" int computes_a_sum () { using namespace cpplinq; int ints[] = {3,1,4,1,5,9,2,6,5,4}; auto result = from_array (ints) >> where ([](int i) {return i%2 ==0;}) // Keep only even numbers >> sum () // Sum remaining numbers ; return result; } Taken from https://archive.codeplex.com/?p=cpplinq https://archive.codeplex.com/?p=cpplinq Done with C++11. I only referred C++17, because of the lambda improvements done since they got introduced in C++11.
  - Sean1708 9y ago
    > list-comprehensions are faster than loops Are they really though? In my experience you'll gain a couple of percent because you're getting rid of that call to append, but that's hardly the orders of magnitude OP was looking for.
    - joshuamorton 9y ago
      Yes, I wasn't saying they would fix this problem (they won't), but making the broader point that in general, more idiomatic/pleasant looking code is faster. This is neither of those, and so my hunch is that there are improvements that can be made to both form and function.
- psyklic 9y ago
  Confirmed. Earlier today I rewrote some (non-NumPy) Python numerical sampling code in C and it was a 100x speedup.
- e12e 9y ago
  > Finally, I've always been able to get Cython to parity with C++. However, when I'm done, I wonder what I gained. The C++ isn't that much more complicated than adequately type annotated Cython. Depends on who needs to work with the code; but sometimes, it might be nice to have an obviously correct (but slow) pure python version, that can share apis and tests with the convoluted "fast enough" version.

civility 9y ago

> I see a 4x speedup on most of my slowest pure python workloads Heh, only 25..250 X to go. We did a direct line for line translation of some numerically intensive code from Python to C++ and saw a literal 1000X speedup. On other projects, it's been more like 100X slower. That says two things: first Python can be really slow, second, for some programs, Python doesn't really save on lines of code over modern C++. I've been very impressed with PyPy however. In testing, it can sometimes sneak up to less than a factor of 2 slower than C. However, the bummer comes when it doesn't hit that mark and you have no idea how to trick the JIT to do better. If it works, great. If it doesn't, you don't have much insight into why. Finally, I've always been able to get Cython to parity with C++. However, when I'm done, I wonder what I gained. The C++ isn't that much more complicated than adequately type annotated Cython.

joshuamorton 9y ago
While I'm more of a pythonist than a C-ist, hearing "1000x speedup" and "line for line" to me implies that you aren't writing idiomatic python. Idiomatic python is (often) faster than not, and (often) more difficult to translate to lower level languages. As a simple example, list-comprehensions are faster than loops, and can't be line for line translated into C++.
- pjmlp 9y ago
  Depends, I bet they can be easily translated to some LINQ like implementation in C++17. List comprehensions are just syntax sugar for map/filter/fold.
  - joshuamorton 9y ago
    A jaunt through SO didn't give anything reasonable. This is what SO had to offer: https://stackoverflow.com/questions/36339533/how-to-generate-vector-like-list-comprehension/36340883 https://stackoverflow.com/questions/36339533/how-to-generate..., and a quick check of new C++17 features didn't show any that would obviously improve on that. There's nothing implicitly stopping you from doing some macro magic to implement it, but its not there naturally.
    - pjmlp 9y ago
      SO is not the only source of truth. #include "cpplinq.hpp" int computes_a_sum () { using namespace cpplinq; int ints[] = {3,1,4,1,5,9,2,6,5,4}; auto result = from_array (ints) >> where ([](int i) {return i%2 ==0;}) // Keep only even numbers >> sum () // Sum remaining numbers ; return result; } Taken from https://archive.codeplex.com/?p=cpplinq https://archive.codeplex.com/?p=cpplinq Done with C++11. I only referred C++17, because of the lambda improvements done since they got introduced in C++11.
- Sean1708 9y ago
  > list-comprehensions are faster than loops Are they really though? In my experience you'll gain a couple of percent because you're getting rid of that call to append, but that's hardly the orders of magnitude OP was looking for.
  - joshuamorton 9y ago
    Yes, I wasn't saying they would fix this problem (they won't), but making the broader point that in general, more idiomatic/pleasant looking code is faster. This is neither of those, and so my hunch is that there are improvements that can be made to both form and function.
psyklic 9y ago
Confirmed. Earlier today I rewrote some (non-NumPy) Python numerical sampling code in C and it was a 100x speedup.
e12e 9y ago
> Finally, I've always been able to get Cython to parity with C++. However, when I'm done, I wonder what I gained. The C++ isn't that much more complicated than adequately type annotated Cython. Depends on who needs to work with the code; but sometimes, it might be nice to have an obviously correct (but slow) pure python version, that can share apis and tests with the convoluted "fast enough" version.