3 ms·

I just gonna repost what I've written before about this before: The current state of threading and parallel processing in Python is a joke. While they are stil

I just gonna repost what I've written before about this before: The current state of threading and parallel processing in Python is a joke. While they are still clinging to the GIL and single-core performance, the rest of the world is moving to 32 core (consumer) CPUs. Python's performance, in general, is a crappy[1] and is beaten even by PHP these days. All the people that suggest relying on multiprocessing probably haven't done anything that's CPU and Memory intensive because if you have a code that operates on a "world-state" each new process will have to copy that from a parent. If the state takes ~10GB each process will multiply that. Others keep suggesting Cython. Well, guess what? If I am required to use another programming language to use threads, I might as well go with Go/Rust/Java instead and save the trouble of dabbling with two languages. So where does that leave (pure-)Python? It can only be used in I/O bound applications where the performance of the VM itself doesn't matter. So it's basically only used by web/desktop applications that CRUD the databases. It's really amazing that the machine learning community has managed to hack around that with C-based libraries like SciPy and NumPy. However, my suggestion would be to drop GIL and copy the whatever model has been working for Go/Java/C#. If you can't drop GIL because some esoteric features depend on that, then drop them as well. [1] https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/python.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

kortex 7y ago
Every single project which has tried to drop the GIL has failed in some way. It's not some "esoteric features", it's fundamentally a hard problem that implicates the entirety of the python object model, python C api, scoping, imports, and GC. I think multi-interpreting is the way to go, but that still would require a framework for ensuring safe memory access. Speaking of Go, I always thought it would be neat to write a python implementation in Go, but leverage Go's GC, and implement the 'go' keyword/function for easy parallelism. But you still have the problem of scoping and memory safety. Or similar idea but with Rust. Something tells me that isn't a trivial undertaking, especially if you want all the libraries, which is 75% the point of python.
bildung 7y ago
> All the people that suggest relying on multiprocessing probably haven't done anything that's CPU and Memory intensive because if you have a code that operates on a "world-state" each new process will have to copy that from a parent. If the state takes ~10GB each process will multiply that. This is wrong, there are multiple ways of python threads working on shared data. > It's really amazing that the machine learning community has managed to hack around that with C-based libraries like SciPy and NumPy. Well, the main implementation of the whole language is c-based. I can't see how that implies hackiness. > If you can't drop GIL because some esoteric features depend on that, then drop them as well. There have been multiple python implementations without a GIL available for way over 10 years, for example pypy, ironpython and jython. Yet these never went mainstream, which strongly implies the GIL problem actually isn't that much of a problem in the real world.

kortex 7y ago

Every single project which has tried to drop the GIL has failed in some way. It's not some "esoteric features", it's fundamentally a hard problem that implicates the entirety of the python object model, python C api, scoping, imports, and GC. I think multi-interpreting is the way to go, but that still would require a framework for ensuring safe memory access. Speaking of Go, I always thought it would be neat to write a python implementation in Go, but leverage Go's GC, and implement the 'go' keyword/function for easy parallelism. But you still have the problem of scoping and memory safety. Or similar idea but with Rust. Something tells me that isn't a trivial undertaking, especially if you want all the libraries, which is 75% the point of python.

bildung 7y ago

> All the people that suggest relying on multiprocessing probably haven't done anything that's CPU and Memory intensive because if you have a code that operates on a "world-state" each new process will have to copy that from a parent. If the state takes ~10GB each process will multiply that. This is wrong, there are multiple ways of python threads working on shared data. > It's really amazing that the machine learning community has managed to hack around that with C-based libraries like SciPy and NumPy. Well, the main implementation of the whole language is c-based. I can't see how that implies hackiness. > If you can't drop GIL because some esoteric features depend on that, then drop them as well. There have been multiple python implementations without a GIL available for way over 10 years, for example pypy, ironpython and jython. Yet these never went mainstream, which strongly implies the GIL problem actually isn't that much of a problem in the real world.