4 ms·

I'm not a very heavy user of Python, but I have used it for a few projects. I've never hit the need to have multiple threads in a Python project, but I wonder:

by CrLf 16y ago

I'm not a very heavy user of Python, but I have used it for a few projects. I've never hit the need to have multiple threads in a Python project, but I wonder: Is the GIL that much of a bottleneck? Isn't the multiprocessing module and its locking primitives enough for most work where CPU-bound processing needs to be distributed over multiple cores? When that fails can't that part of the software be done in C with Python as a wrapper language? I see people here complaining about how the GIL limits their use of Python for serving requests and whatnot. Stuff that looks like the canonical example of something that can be solved by a multi-process module. But I also see people complaining about stuff that is really CPU-bound, without seeing them state that they already overcome the main problem with this stuff, which is making them (theoretically) parallel in the first place. I'm from an unix background, where threading is as heavy (or as lightweight) as multi-processing. Maybe much of the complainers come from Windows, where the equivalent of fork is very heavy compared to threading. Maybe the problem is under Python, and not really the GIL. I'm not saying that the GIL is not a problem. But if it hasn't been removed yet, maybe the problems that come from removing it are even worse. And Python isn't the only tool around. You can combine languages in a single project...

ramchip 16y ago
I tend to use threads for simple I/O cases, not as an optimization per se. For example, I have an image processing library thanks to which I can do a simple image.save(), but since I don't want to sit idle for milliseconds until it returns I have a thread pool run it. Same thing if I have an expensive operation and I want the GUI to keep responding. Then again, I'll confess that I am not much of a Python user either and multiprocessing may be a sufficient answer.
jnoller 16y ago
As the maintainer of multiprocessing, I can say I've never really been bothered by the GIL. I know about it - I know that if I have CPU bound tasks, I'm going to better off using processes, ergo, multiprocessing. I spin up some processes, pop in a queue and I'm off the the races. That said - the majority of my code uses plain old python threads. Web load testing tools, subprocess execution, etc, etc - anything with I/O works fine for me contained in threads and since thats where I spend most of my time (in I/O) they work fine for me. In my last "multiprocessing heavy" chunk of code, I wasn't using it for local processes and work-sharing. I was using it to spread work over a network of hosts using managers and the other tools within it. The one gotcha with jumping in between the two is serialization. When you deal with multiprocessing, your objects which are passed in between the pools/queues/processes much be pickle-able - this means for tasks which contain lots of unpickle-able, shared state multiprocessing is not a good answer. Essentially, having free threading in cPython would mean that you could have your cake (concurrency on multiple cores) and eat it (without incurring serialization, mutable shared state) too.

by CrLf 16y ago

ramchip 16y ago

I tend to use threads for simple I/O cases, not as an optimization per se. For example, I have an image processing library thanks to which I can do a simple image.save(), but since I don't want to sit idle for milliseconds until it returns I have a thread pool run it. Same thing if I have an expensive operation and I want the GUI to keep responding. Then again, I'll confess that I am not much of a Python user either and multiprocessing may be a sufficient answer.

jnoller 16y ago

As the maintainer of multiprocessing, I can say I've never really been bothered by the GIL. I know about it - I know that if I have CPU bound tasks, I'm going to better off using processes, ergo, multiprocessing. I spin up some processes, pop in a queue and I'm off the the races. That said - the majority of my code uses plain old python threads. Web load testing tools, subprocess execution, etc, etc - anything with I/O works fine for me contained in threads and since thats where I spend most of my time (in I/O) they work fine for me. In my last "multiprocessing heavy" chunk of code, I wasn't using it for local processes and work-sharing. I was using it to spread work over a network of hosts using managers and the other tools within it. The one gotcha with jumping in between the two is serialization. When you deal with multiprocessing, your objects which are passed in between the pools/queues/processes much be pickle-able - this means for tasks which contain lots of unpickle-able, shared state multiprocessing is not a good answer. Essentially, having free threading in cPython would mean that you could have your cake (concurrency on multiple cores) and eat it (without incurring serialization, mutable shared state) too.