6 ms·

even python with asyncio and multithreading / multiprocessing

armitron 7y ago
Python with asyncio and multithreading doesn't take good advantage of multiple cores due to the global interpreter lock. With multiprocessing, one pays big costs in IPC. Python is not at all suitable today for parallelism. Which is one reason why languages like Go and Elixir are gaining so much traction.
- no_wizard 7y ago
  They’re actively working on a neat angle at solving this problem via much cleaner IPC through spawning sub interpreters https://www.python.org/dev/peps/pep-0554/ https://www.python.org/dev/peps/pep-0554/ This just may be the way forward in the Python ecosystem
  - blattimwind 7y ago
    Subinterpreters share a common GIL. Also subinterpreters don't share Python objects, which means among other things that all modules are imported separately in each SI, which increases startup time, memory usage and reduce cache effectiveness. It's a band-aid. If you want to run Python code in parallel, without large overhead, then CPython is simply not your environment to do so, and Python is not a good choice overall in that kind of endeavour.
- blattimwind 7y ago
  > Python is not at all suitable today for parallelism. As a long time Python dev, including work on parallel applications, I have to agree. It's always annoying in Python, and entirely Un-Pythonic.
montecarl 7y ago
Or python with mpi4py. MPI is the perfect multiprocessing paradigm for parallel python code, since you avoid the GIL. You can easily use MPI on a single workstation, or scale it to run on a supercomputer.
MaxBarraclough 7y ago
Python is just about the worst major programming language to write high-performance code in. The only slower major language is Ruby. If you're using Python to invoke highly-optimised native-code, then your performance will be excellent (as shown by the various Python numerical libraries), but performance-sensitive code shouldn't run in the Python interpreter. As others have said, Python also lacks true multithreading (its threads are capable of concurrency but not parallelism, on account of the GIL), but you do have the option of just running a bunch of Python processes in parallel. I imagine that's a workable solution at least some of the time, but I've never explored this, so I don't know how good the library support is. Edit: Someone else mentioned 'mpi4py' which seems to be a Python library for multi-process work.

Python with asyncio and multithreading doesn't take good advantage of multiple cores due to the global interpreter lock. With multiprocessing, one pays big costs in IPC. Python is not at all suitable today for parallelism. Which is one reason why languages like Go and Elixir are gaining so much traction.

no_wizard 7y ago
They’re actively working on a neat angle at solving this problem via much cleaner IPC through spawning sub interpreters https://www.python.org/dev/peps/pep-0554/ https://www.python.org/dev/peps/pep-0554/ This just may be the way forward in the Python ecosystem
- blattimwind 7y ago
  Subinterpreters share a common GIL. Also subinterpreters don't share Python objects, which means among other things that all modules are imported separately in each SI, which increases startup time, memory usage and reduce cache effectiveness. It's a band-aid. If you want to run Python code in parallel, without large overhead, then CPython is simply not your environment to do so, and Python is not a good choice overall in that kind of endeavour.
blattimwind 7y ago
> Python is not at all suitable today for parallelism. As a long time Python dev, including work on parallel applications, I have to agree. It's always annoying in Python, and entirely Un-Pythonic.

montecarl 7y ago

Or python with mpi4py. MPI is the perfect multiprocessing paradigm for parallel python code, since you avoid the GIL. You can easily use MPI on a single workstation, or scale it to run on a supercomputer.

MaxBarraclough 7y ago

Python is just about the worst major programming language to write high-performance code in. The only slower major language is Ruby. If you're using Python to invoke highly-optimised native-code, then your performance will be excellent (as shown by the various Python numerical libraries), but performance-sensitive code shouldn't run in the Python interpreter. As others have said, Python also lacks true multithreading (its threads are capable of concurrency but not parallelism, on account of the GIL), but you do have the option of just running a bunch of Python processes in parallel. I imagine that's a workable solution at least some of the time, but I've never explored this, so I don't know how good the library support is. Edit: Someone else mentioned 'mpi4py' which seems to be a Python library for multi-process work.