4 ms·

A perfectly written parallel Python app without GIL would be ideally 8 times faster on an 8 core system, compared to stock Python. Rewriting the original progr

A perfectly written parallel Python app without GIL would be ideally 8 times faster on an 8 core system, compared to stock Python. Rewriting the original program in Java would make the program about 100 times faster... on one core. Now let's see. We deal with hacked Python, with complex error-prone thread synchronization code in Python for 8x gain, or grab the 100x gain in a single thread in Java? While I don't mind Python getting faster, threading should be the last thing to try before everything else is exhausted in order to make Python run fast in one thread.

Derbasti 12y ago
Give me Numpy, Scipy, Matplotlib, Scikit-learn, and iPython in Java and we are talking.
- coldtea 12y ago
  I think you missed his whole point. Whoosh. His point was about using Java instead of Python. It was that at best, a perfect parallel CPython can yield a N times more performant program (where N = number of cores). And that's for a perfect parallel CPython, and only when there's very little sync/sharing overhead and the program is also fully parallelizable. So, even for an 8 core machine, parallelizing Python can yield at best c*8 better performance, where c < 1. Now, what he says is, there are languages that run stuff 10 and 100 times faster than CPython on a SINGLE core. So maybe start from there (improving single core CPython performance), something which has much more room to speed up our code, even if it's not parallel.
  - trentnelson 12y ago
    You're right, even if you're still running efficiently on all cores, it's still Python. I actually make this exact point in a separate presentation that I did a few weeks ago: https://speakerdeck.com/trent/parallelism-and-concurrency-with-python?slide=33 https://speakerdeck.com/trent/parallelism-and-concurrency-wi... Numba is one option -- rather than executing the CPython innards, have it switched out for an LLVM-optimized kernel instead.
  - Shish2k 12y ago
    I think you missed his whole point. Whoosh. His point was about using the language with the most productive / fastest libraries. Single-thread python vs 8-thread python vs java makes no difference if the majority of your processing time is spent inside a highly optimised native library.
    - coldtea 12y ago
      I think you missed my whole point. Whoosh. I (or the OP) didn't tell anybody to use Java instead of Python. What we said is that it will be better for performance to optimize CPython's core speed instead of its multi-core capabilities. So the remark about Python having more libs than Java is beside the point, since Java wasn't mentioned as a migration option, but as an example of how far single-core performance can be taken, and as an advice to try to get some of that to Python. Oh, and the "but still it doesn't matter because Python has fast native libs" is not an argument either, because if that was enough people wouldn't care about parallelizing Python to get more speed. Which was the whole topic that started this thread.
      - Shish2k 12y ago
        I think you missed my whole point. Whoosh. (PS. This is silly :P) > if [native code] was enough people wouldn't care about parallelizing Python They would, for the same reason that they care about parellelizing Java -- once you're hitting the limits of single-thread speed (whether it's by being fast yourself or by having fast extensions), multithreading is next on the list.
    - mantrax5 12y ago
      Coldtea got my point pretty well. Everyone else gets a whoosh. I'm not saying abandon Python for Java, I'm saying if you'll optimize Python, first optimize single-thread Python performance, before we get to parallel processing, because extracting performance out of parallel is hard, while languages like Java (and JavaScript recently) show they there's plenty more to be gained yet from that single thread.
      - 12y ago
PythonicAlpha 12y ago
<irony>Why not implement everything in assembly language and use all parallelizing tricks of e.g. the intel architecture has to offer and get a speed-up of (in some selected cases) 10.000 times? Of course development time will also go up 10.000 times!</irony>

Derbasti 12y ago

Give me Numpy, Scipy, Matplotlib, Scikit-learn, and iPython in Java and we are talking.

coldtea 12y ago
I think you missed his whole point. Whoosh. His point was about using Java instead of Python. It was that at best, a perfect parallel CPython can yield a N times more performant program (where N = number of cores). And that's for a perfect parallel CPython, and only when there's very little sync/sharing overhead and the program is also fully parallelizable. So, even for an 8 core machine, parallelizing Python can yield at best c*8 better performance, where c < 1. Now, what he says is, there are languages that run stuff 10 and 100 times faster than CPython on a SINGLE core. So maybe start from there (improving single core CPython performance), something which has much more room to speed up our code, even if it's not parallel.
- trentnelson 12y ago
  You're right, even if you're still running efficiently on all cores, it's still Python. I actually make this exact point in a separate presentation that I did a few weeks ago: https://speakerdeck.com/trent/parallelism-and-concurrency-with-python?slide=33 https://speakerdeck.com/trent/parallelism-and-concurrency-wi... Numba is one option -- rather than executing the CPython innards, have it switched out for an LLVM-optimized kernel instead.
- Shish2k 12y ago
  I think you missed his whole point. Whoosh. His point was about using the language with the most productive / fastest libraries. Single-thread python vs 8-thread python vs java makes no difference if the majority of your processing time is spent inside a highly optimised native library.
  - coldtea 12y ago
    I think you missed my whole point. Whoosh. I (or the OP) didn't tell anybody to use Java instead of Python. What we said is that it will be better for performance to optimize CPython's core speed instead of its multi-core capabilities. So the remark about Python having more libs than Java is beside the point, since Java wasn't mentioned as a migration option, but as an example of how far single-core performance can be taken, and as an advice to try to get some of that to Python. Oh, and the "but still it doesn't matter because Python has fast native libs" is not an argument either, because if that was enough people wouldn't care about parallelizing Python to get more speed. Which was the whole topic that started this thread.
    - Shish2k 12y ago
      I think you missed my whole point. Whoosh. (PS. This is silly :P) > if [native code] was enough people wouldn't care about parallelizing Python They would, for the same reason that they care about parellelizing Java -- once you're hitting the limits of single-thread speed (whether it's by being fast yourself or by having fast extensions), multithreading is next on the list.
  - mantrax5 12y ago
    Coldtea got my point pretty well. Everyone else gets a whoosh. I'm not saying abandon Python for Java, I'm saying if you'll optimize Python, first optimize single-thread Python performance, before we get to parallel processing, because extracting performance out of parallel is hard, while languages like Java (and JavaScript recently) show they there's plenty more to be gained yet from that single thread.
    - 12y ago

PythonicAlpha 12y ago

<irony>Why not implement everything in assembly language and use all parallelizing tricks of e.g. the intel architecture has to offer and get a speed-up of (in some selected cases) 10.000 times? Of course development time will also go up 10.000 times!</irony>