4 ms·
Ask HN: Overcoming the GIL in Python
I've been working on a python app that I've been trying to multithread for efficiency. The Global Interpreter Lock is preventing me from using the built in threading module to increase speed.
I've done some searching and discovered the multiprocessing module (formerly processing) which has a similar API to threading except that it spawns multiple interpreter processes to overcome the GIL. I've also read that it is possible to write a C module that can release the GIL and then reacquire it, but I would like to avoid this if possible. I'm working on a simple physics engine for fun, and I think that writing it in C might take the fun right out of it.
- deleted 17y ago[deleted]
- russell 17y agoI have been interested in this issue for a long time, but haven't dug into it enough to be authoritative. The GIL isn't going to be fixed until someone rewrites the interpreter, a task for a genius with time on his hands. I don't think managing the GIL in C will do what you want. If it were easy, someone would have done it. If I were you, I would go go with a package that implements a good message passing model a la Erlang. It would be a good experience. Alternatively, you could look at Jython or Iron Python which are built on top of vms with native thread support.
- stonemetal 17y agoYep, all true. You may want to check out scipy. It provides a good base for numerical integration and other things you are going to need. It is written in C so perhaps you won't have to drop into C for performance reasons.
- mcav 17y agoAlso check out unladen-swallow -- they aim to remove the GIL from Python (among other goals): http://code.google.com/p/unladen-swallow/wiki/ProjectPlan http://code.google.com/p/unladen-swallow/wiki/ProjectPlan
- jncraton 17y agoThat looks like it could eventually be incredibly useful. I hope that they write their code in a way which allows some of their improvements to be added to Python.
- jaydub 17y agoThis may be of interest: "Stackless Python allows you to use lightweight threads (tasklets) that can be switched with less overhead and it allows for cooperative multitasking with the intention of making async programming easier. This helps with IO bound applications where you can have tens of thousands of tasklets running at the same time (try doing that with threads). It does not however allow you to take advantage of multi-core or multi-processor resources. It simply allows you to squeeze the most amount of work out of a single threaded process..." http://www.stackless.com/pipermail/stackless/2007-August/001963.html http://www.stackless.com/pipermail/stackless/2007-August/001...
- jncraton 17y agoI will probably end up using stackless. CCP seems to have gotten it to work quite well for EVE. I'm still really impressed by their single server MMO architecture.
- wmf 17y agoStackless is the opposite of what he wants, which is to take advantage of multi-core or multi-processor resources.
- jncraton 17y agoIt turns out you are absolutely correct. Thanks for pointing that out. "Stackless python does not make use of any kind of multi-core environment it runs on." http://stackoverflow.com/questions/377254/stackless-python-and-multicores http://stackoverflow.com/questions/377254/stackless-python-a...
- jaydub 17y agoStackless is the opposite of what he wants The quote above does mention that "[Stackless] does not however allow you to take advantage of multi-core or multi-processor resources. It simply allows you to squeeze the most amount of work out of a single threaded process...". Depending on the application and its particular implementation, actually Stackless could be the right solution. Which is why I brought it up. In fact, it now looks as though he might go with Stackless, just like EVE online did :)