4 ms·
So Pythonistas, you want to get rid of the GIL...
- lmm 14y agoDo actual pythonistas really hate the GIL? I thought it was mostly a troll talking point, just like the whitespace.
- jforman 14y agoWhy would the inability to write multi-threaded programs that are worth a damn be a troll talking point? It meaningfully lowers the appropriateness of using python for server code.
- nakkiel 14y agoThere's more than one way to do it. If you're thinking about hundreds of threads doing chunks of work, then my comment will not be meaningful but if you're talking about maximizing CPU utilization for a particular task, then multiple processes is a valid and proven approach (Tornado does just that for example, forking as many processes as you have CPUs and putting them to work).
- jedbrown 14y agoThe viability of separate processes is highly dependent on the problem domain and algorithm. Threads have their place.
- gaius 14y agoNo, the GIL is simply not relevant for real work. True Pythonistas, when performance is critical, just drop into C for the crucial section and use Python to control it. When C code is running, the Python runtime isn't blocked - you can kick off a hundred native pthreads in the background if you want to. This is also true in OCaml BTW, but it's trickier to make the call as when running single-threaded, OCaml code is as fast as C anyway. A Python/OCaml hybrid using C as glue is probably the best of both worlds.
- tetha 14y agoor you look at multiprocessing, which spreads the computation into multiple processes, while enforcing clear boundaries and communication methods.
- anacrolix 14y agoIt's becoming a problem. Even if your code isn't CPU bound, the GIL causes terrible scheduling problems.
- rogerbinns 14y agoThe main problem is the perception that code would linearly scale if the GIL was removed. Eg on a 4 core machine code would run 4 times as fast. And that the GIL has no value. Both statements are completely false. (A bigger issue is the reference counting anyway. Changing that should be a precursor to GIL removal.) I'm an author of a C extension (apsw) and the GIL is fantastic. Without it the code would be considerably more complicated, longer and likely to have race conditions and similar gnarly bugs. As a Python developer I use: * Multiple processes usually by executing a new process which allows me to easily substitute testing processes and using stdin/stdout * Use networking for talking between parts (allows spreading over multiple machines, failover, redundancy, load balancing etc) * multiprocessing for pure Python work * Threads for I/O bound work or work that waits on other things * C extensions that release the GIL Actually getting rid of the GIL wouldn't help.
- amirmansour 14y agoI'm actually interested to know what the HN community does to achieve parallelism in Python? Do you use multiprocessing module, Stackless python, PP module (parallel python)?
- willvarfar 14y agoI use workers (sometimes multiprocessing, sometimes I do my own starting of them). Often I have use-cases where my own shared memory system works well http://williamedwardscoder.tumblr.com/post/13363076806/buffcacher http://williamedwardscoder.tumblr.com/post/13363076806/buffc... > I was writing a custom video editing program in Python (interesting choice of language for that problem) and I wanted to cache decoded frames
- dagw 14y agoEither multiprocessing or shifting the bits that really benefit from parallelization out to C/C++ tend to cover the vast majority of my needs. Also numpy and numexpr does a lot of the multiprocessing stuff for me, so in many cases I don't even have to think about it.
- deleted 14y ago[deleted]
- dbaupp 14y agoFor sufficiently new version of Python (3.2+), concurrent.futures[1] is pretty nice for its use-case, but otherwise multiprocessing has been quite good. [1]: http://docs.python.org/py3k/library/concurrent.futures.html http://docs.python.org/py3k/library/concurrent.futures.html
- blitzprog 14y agoI'm using QThread for PyQt GUI applications. However the GIL makes it impossible to execute _Python_ code in parallel. I can understand the reasons for that, however it doesn't change the fact that it makes my application inherently slower and is therefore something I dislike. Apart from that I really like Python.
- 14y ago
- cpeterso 14y agoPython objects only need to be thread-safe if they are accessed by multiple threads. A proven solution that could preserve Python's atomicity guarantees is actor-based concurrency like Erlang or Go. When Python objects are queued to an actor's channel, a deep copy (or copy-on-write reference) could be sent to avoid data sharing.
- earl 14y agoThe blog's example is stupid. Of course if you pop a list in more than one thread the behavior is undefined. Who on earth doesn't expect that? @jagw: ok fair, but that is going to be the very first example of things thou shalt not do when using threads in any article, book, or course
- dagw 14y agoPeople who have never done proper multi-threaded programming.
- sausagefeet 14y agoIt's defined in Python.
- comex 14y ago> You will rewrite your code to not make use of the guarantees. I suppose a problem is that unlike Java, Python is not fast enough to implement low level data structures in the language itself, so you can't easily implement non-thread-safe data structures without losing memory safety at the C level, which is unacceptable. Still, I have the feeling that with heavy use of biased locking, you could stick locks everywhere without a lot of overhead in the non-contended case. Or maybe that famous patch just needs to be revisited, tweaked to more efficiently address the areas of highest contention... see http://dabeaz.blogspot.com/2011/08/inside-look-at-gil-removal-patch-of.html http://dabeaz.blogspot.com/2011/08/inside-look-at-gil-remova... for some context. (some forms of biased locking are patented.)
- xlevus 14y agoCould we not have a per-object lock, but only on objects the programmer defines as requiring thread safety?
- jbert 14y agoIf you tag objects as being either "owned" by a thread or "shared" then you could (in the python implementation) do something like: if (obj.owner == current_thread): # just get in and mess with it elsif (obj.owner == shared): # get in and mess with it under a lock else throw "not your obj - make this obj shared if you need that" From a language perspective, you'd just need a way to declare that an object was 'shared'. All other objs would be owned by their creating thread.
- anacrolix 14y agoIt's not that easy.
- exDM69 14y agoNo, that doesn't work. Java has something like this (synchronized methods) but they are cumbersome (I find programming nicer with plain old locks and conditions, the Java synchronized model feels just a layer of cruft but its all lock&cond inside). But Python is different than Java. The Python interpreter has to guard the internal data structures of the interpreter itself, and that is a lot more tricky than it seems on the surface.