3 ms·

> asyncio ...is useless for CPU bound tasks. The event loop uses only one core. > multiprocessing ...relies on IPC and running actual system processes, both

> asyncio ...is useless for CPU bound tasks. The event loop uses only one core. > multiprocessing ...relies on IPC and running actual system processes, both of which have alot more overhead than switching thread context and using shared memory. > For any use case where the last few percent matter, consider not using Python (which will be much much more significant). Here is an interesting question: If asyncio and multiprocessing already give us "nearly the same or better performance", then why is "use another language" such a common advice to escape parallelism-problems in Python? Because, curiously enough, the languages that are usually recommended for this (C, Go, Rust, C++, Java) all implement thread-based parallelism.

n2d4 3y ago
If you're interested, I wrote a more elaborate comment on asyncio vs. multiprocessing vs. multithreading here: https://news.ycombinator.com/item?id=36915581 https://news.ycombinator.com/item?id=36915581 (fyi, Python's multiprocessing has support for interprocess shared memory, with overhead) Those languages you mentioned are not only recommended to escape parallelism problems in Python. They are recommended because they are much, much faster, period. Four of those you listed compile down to machine code, the last one has a highly optimized JIT. Just two are garbage-collected, and none do reference counting. All of them are strongly typed. These differences sum up to orders of magnitudes. If well-written Python code were equally fast as well-written C code but the only way to do parallelism was using processes, then I promise you you wouldn't get (most of) those voices telling you to escape Python. Plenty of well-written C programs choose a multiprocessing approach for parallelism over multithreading. Long story short, if you worry about the performance overhead between multithreading and multiprocessing, make sure you worry about plenty of more significant factors that differentiate Python from faster languages first.
- usrbinbash 3y ago
  > (fyi, Python's multiprocessing has support for interprocess shared memory, with overhead) I know. I have a werkzeug/gunicorn application that currently runs 60 worker processes on a 64 core server. And I would love nothing better than to rip out every. single. last. one. of the IPC facilities, and replace them with the same easy and convenient mutex and CSP systems, that I can use in my Golang applications. > They are recommended because they are much, much faster, period. I know, but "rewrite it in Rust/Go/C++" isn't always an option, be it because of legacy status, constraints in available dev-hours, compatibility problems, library support or simply ease of use.

n2d4 3y ago

If you're interested, I wrote a more elaborate comment on asyncio vs. multiprocessing vs. multithreading here: https://news.ycombinator.com/item?id=36915581 https://news.ycombinator.com/item?id=36915581 (fyi, Python's multiprocessing has support for interprocess shared memory, with overhead) Those languages you mentioned are not only recommended to escape parallelism problems in Python. They are recommended because they are much, much faster, period. Four of those you listed compile down to machine code, the last one has a highly optimized JIT. Just two are garbage-collected, and none do reference counting. All of them are strongly typed. These differences sum up to orders of magnitudes. If well-written Python code were equally fast as well-written C code but the only way to do parallelism was using processes, then I promise you you wouldn't get (most of) those voices telling you to escape Python. Plenty of well-written C programs choose a multiprocessing approach for parallelism over multithreading. Long story short, if you worry about the performance overhead between multithreading and multiprocessing, make sure you worry about plenty of more significant factors that differentiate Python from faster languages first.

usrbinbash 3y ago
> (fyi, Python's multiprocessing has support for interprocess shared memory, with overhead) I know. I have a werkzeug/gunicorn application that currently runs 60 worker processes on a 64 core server. And I would love nothing better than to rip out every. single. last. one. of the IPC facilities, and replace them with the same easy and convenient mutex and CSP systems, that I can use in my Golang applications. > They are recommended because they are much, much faster, period. I know, but "rewrite it in Rust/Go/C++" isn't always an option, be it because of legacy status, constraints in available dev-hours, compatibility problems, library support or simply ease of use.