6 ms·
The current state of threading and parallel processing in Python is a joke. While they are still clinging to the GIL and single core performance, the rest of th
by tasubotadas 7y ago
The current state of threading and parallel processing in Python is a joke. While they are still clinging to the GIL and single core performance, the rest of the world is moving to 32 core (consumer) CPUs.
Python's performance, in general, is a crappy[1] and is beaten even by PHP these days. All the people that suggest relying on multiprocessing probably haven't done anything that's CPU and Memory intensive because if you have a code that operates on a "world-state" each new process will have to copy that from a parent. If the state takes ~10GB each process will multiply that.
Others keep suggesting Cython. Well, guess what? If I am required to use another programming language to use threads, I might as well go with Go/Rust/Java instead and save the trouble of dabbling with two languages.
So where does that leave (pure-)Python? It can only be used in I/O bound applications where the performance of the VM itself doesn't matter. So it's basically only used by web/desktop applications that CRUD the databases.
It's really amazing that the machine learning community has managed to hack around that with C-based libraries like SciPy and NumPy. However, my suggestion would be to drop GIL and copy the whatever model has been working for Go/Java/C#. If you can't drop GIL because some esoteric features depend on that, then drop them as well.
[1] https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/python.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- deleted 7y ago[deleted]
- AlexTWithBeard 7y agoCython is nice, but debugging it requires gdb. For the PyCharm-loving end-users it may be quite cumbersome. Those recommending to use multiprocessing have probably never been in that bitter spot when serializing something and computing something takes exactly the same time. Also forking didn't really work until Python 3.6.
- juststeve 7y ago> Go/Rust/Java And there's also Kotlin.
- gray_-_wolf 7y ago> If the state takes ~10GB each process will multiply that. In POSIX there is such thing as copy-on-write memory during forks.. So if that state is mostly read-only, additional memory required by each slave should be minimal.
- pjmlp 7y agoThere is no such thing, as COW on fork() is implementation specific, although most UNIXes do follow it.
- gray_-_wolf 7y agoHeh, my bad, did not know. Is there any significant UNIX system that does not do COW?
- pjmlp 7y agoProbably not, as it is a common optimization. However it isn't required by POSIX for compliance. http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html http://pubs.opengroup.org/onlinepubs/9699919799/functions/fo... http://pubs.opengroup.org/onlinepubs/9699919799/ http://pubs.opengroup.org/onlinepubs/9699919799/
- dual_basis 7y agoI hereby award you 1 pedantic point. Spend it sensibly!
- dual_basis 7y agoThe consistent requirement has been that Python will drop the GIL for anything that doesn't make single-threaded performance suffer. There has been substantial work to this end but no solution to date has achieved this goal.