34 ms·
Let's Remove the Global Interpreter Lock
- vasilakisfil 9y agoJust curious: if they solve it in Python, would it be possible to solve it in Ruby too ?
- taf2 9y agoIf I understand correctly the issue in Ruby is the existing C extensions that have been written to assume the lock exists...
- myusernameisok 9y agoIt's the same issue with Python. AFAIK there are a number of Python libraries that are not thread-safe, and the GIL prevents them from being an issue.
- shwouchk 9y agoGIL does not help thread safety in application code (and external libraries), just in the VM.
- myusernameisok 9y agoI'm fairly certain you're incorrect. With the GIL you don't have to lock shared memory because the assumption is that only one thread will be running at a time. For example shared data structures won't be changed while being being read/written to by multiple threads, because only one thread is actually running.
- shwouchk 9y agoYou are entirely mistaken, unless all you care about are the basic built in dict/list and some of the other built in data structures AND each thread only stores OR reads data (i.e. never reads and then stores it again), from a SINGLE container (you never care about consistent state between two different objects). In my experience this is almost never the case. Moreover, this type of synchronization is trivial to accomplish with relatively little performance sacrifice. What is much more complicated is getting more complex logic work correctly and performantly when you are interacting with multiple different data structures from something more than a saturated loop.
- aidenn0 9y agoI thought the GIL was not held during execution of foreign code in python (at least that was one point given for why the GIL wasn't a big deal in practice).
- dom0 9y agoNo, it must be explicitly released. The GIL must be held to invoke almost all Python runtimes (main exceptions: acquiring the GIL, low-level allocator).
- fny 9y agoThis is in the cards for Ruby 3. The plan is to migrate away from a global interpreter lock to "guilds" which are akin to execution contexts. These guilds also have a locking mechanism that allows for parallelism which they call the "global guild lock." You can learn more about concurrency in Ruby 3 at this wonderful blog post: http://olivierlacan.com/posts/concurrency-in-ruby-3-with-guilds/ http://olivierlacan.com/posts/concurrency-in-ruby-3-with-gui...
- pjmlp 9y agoIt is already solved for Ruby when you use JRuby.
- chrisseaton 9y agoJRuby, Rubinius and TruffleRuby are all existing implementations of Ruby without a GIL.
- dfox 9y agoFor this environment (ie. you already have some kind of concurrent GC) it is probably significantly easier problem for Ruby than for Python. Ruby does not have that much dynamism compated to "everything is dict of pointers to dicts" Python.
- benhoyt 9y agoExcellent! Where's the Donate button or call to action for businesses who want to support this? There's a small link in the sidebar to "Donation page", but that doesn't seem to have a place to donate for the remove-the-GIL effort.
- Herald_MJ 9y ago> Since such work would complicate the PyPy code base and our day-to-day work, we would like to judge the interest of the community and the commercial partners to make it happen (we are not looking for individual donations at this point).
- fijal 9y agoAs mentioned in the blog post the individual donation buttons are not a resounding success. I'm happy to sign contracts with corporate donors (or even individuals) that we'll deliver. My mail should be public, if not #pypy on freenode or fijal at baroquesoftware.com
- btown 9y agoIs the issue that individual donations are unpredictable (and therefore difficult to use as justification for such a large scope increase)? Would you consider setting up something akin to a Patreon to allow individuals to commit to recurring monthly support for the project?
- fijal 9y agoThe main issue is that the effort it takes to setup and maintain it greatly outweighs the amount of money we get (typically). There is also complexity with taxation, jurisdictions and all kinds of mess that is usually very much not worth couple dollars (e.g. $7/week on gratipay for example)
- tyingq 9y agoWould this work with cpython extensions that were ported to PyPy?
- fijal 9y agoThey would run under GIL (I can't see CPython C API being thread-friendly unless gilectomy succeeds)
- tyingq 9y agoAh, ok. So this approach doesn't completely remove the GIL, but removes it as a barrier for pure python code running in PyPy? Or does it break the current support for porting cpython extensions?
- fijal 9y agoIt removes it for pure python code. The C extensions run under the lock (which is unfair to call interpreter any more)
- wyldfire 9y ago> We have some money left in the donation pot for STM which we are not using; according to the rules, we could declare the STM attempt failed and channel that money towards the present GIL removal proposal. I didn't donate to that pot but that does seem like a judicious and reasonable step to take given the assessment of STM.
- macrael 9y agoDo people here use pypy in production? What are the benefits?
- sillysaurus3 9y agoSure. Free 2-5x speedup. pypy + pypy's pip generally works as a transparent drop-in replacement to python + python's pip, so it's free speed. It doesn't (or didn't) work when you need to rely on an extension that uses Python's C API. I haven't followed the scene in awhile so maybe that's changed. pypy's pip has so many libraries that I hardly notice, so maybe they solved that. Unfortunately python is fundamentally slower than lua or JS, possibly due to the object model. Python traps all method calls, but even integer addition, comparisons, and so on are treated as metamethods. That's the case for Lua too, but e.g. it's absurdly easy to make a Python object have a custom length, whereas Lua didn't have a __len__ metamethod until after 5.1. I'm not sure it even works on LuaJIT either. Probably in the newer versions.
- JulianWasTaken 9y agoI can't tell what you mean by the last paragraph there, but oftentimes PyPy's speedups come exactly from inlining stuff like what you refer to there -- Python's not fundamentally slower, it's those kinds of stuff that you can speed up. (And yeah the CPython API is still a pain point if you've got a library that uses it, although some stuff will still work using PyPy's emulation layer. It'd be great if people stopped using it though.)
- sillysaurus3 9y agoFor example, Python makes it fairly easy to trap a call to a missing method, both via __getattr__ and __missing__. In JS the only way you can do that is via Proxy objects, and even those have limits. You can't always inline the arithmetic ops effectively. You can recompile the method each time it's called with different types, but that's why the warmup time is an issue. This wouldn't be a problem if Python didn't make it so trivial to overload arithmetic. JS doesn't.
- dec0dedab0de 9y agoCould someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or working off a queue. #4 does sound appealing, but is it really worth the effort?
- make3 9y agoThere are many cases where the objects are too big to be passed around. Python is used a huge amount in Machine learning and datascience, where being able to do parallel work on stuff already in memory would be great.
- leereeves 9y agoAre those applications often bottlenecked by the CPU, as opposed to GPU or data transfer?
- rspeer 9y agoThe world of algorithms that run well on a CPU is still much, much bigger than the world of algorithms that run well on a GPU, even in machine learning. And even if you're fortunate enough that Nvidia designs their GPUs to solve your problem, why should the CPU cores sit idle?
- dec0dedab0de 9y agoSo work on data that can not be broken down into smaller chunks? That makes sense, and is something I never come across.
- chrisseaton 9y agoI'm sure they can be broken down into smaller chunks, but is it more efficient if they aren't broken down and instead shared memory is used? If you want parallelism you're obviously already worried about performance.
- devwastaken 9y agoThis would be great if it means we can run the C portions of Python in threads without performance hits. I recently started a little project that is a cross-platform GUI for batch bzip2 compression, and Python did it quite well with its built-in bzip2 module. But, once I tried to do it parallel, the performance impacts of GIL were obvious. Yes, you can work around that with multi-process, but I'd rather not be spamming the running processes list and have to actually handle seperate processes that should be threads. In the end I settled for C++ and QT with the native bzip2 library with a few modifications.
- chocolatebunny 9y agoWere you running on Windows or Linux? It's my understanding that multiple processes doesn't have a big performance penalty on Linux compared to multiple threads.
- devwastaken 9y agoI was running under Windows, but the application is meant for Windows linux and mac.
- orf 9y agoDid you investigate the multiprocessing library? Also this kind of thing should be relatively light on the GIL if done correctly. The bzip2 module releases the GIL (I assume?), as does file IO, which is most of the workload in your use case?
- apenwarr 9y agoIn normal CPython, you can design your C extension (such as bzip2) to release the GIL while it runs. This is one of the few times when threads are useful in Python. It's also why scipy etc are as fast as they are. I don't know if the bzip2 module does this, but it probably should.
- akx 9y agoYeah, the stdlib bz2 module does release the GIL for the time of the compression (though it locks the compressor object simultaneously): https://github.com/python/cpython/blob/d4b93e21c2664d6a78e0656e7a7be0807be1c352/Modules/_bz2module.c#L242 https://github.com/python/cpython/blob/d4b93e21c2664d6a78e06...
- zzzeek 9y agoSuper glad they're going to try using mutexes and not that STM approach which was looking to be immensely complicated. Was not looking forward to the kinds of interpreter bugs that was going to produce.
- Beltiras 9y agoThink about a recursive function whose implementation is changed while it is running. The replacement might have an entirely different algorithm. Which version finishes the stack call?
- chrisseaton 9y agoThe version that was originally activated. I think that's the case in every single parallel implementation of a programming language ever. I can't imagine it working any other way. When you redefine a method in any language I'm aware of you just change which method the name points to. You don't modify the original method.
- Beltiras 9y agoSo a function: def fun(*args): if not args: return 0 return fun(*(args[1:])) would be call-by-address after the first invocation? It could be lookup-by-name by way of code.
- chrisseaton 9y agoThe naive implementation, and the semantic model, is always lookup-by-name on every invocation. In practice we apply speculative optimisations including inline caching and guard removal with remote dynamic deoptimisation via safe points to make it a direct call instead.
- seunosewa 9y agoThe ideal solution is for someone to design a new programming language that is as similar to Python as possible without requiring a global lock. Rarely used features that make it hard to parallelize Python would be dropped. STM might be built into the language instead of being hacked into one implementation, etc.
- jrs95 9y agoPerl 6 doesn't have a GIL, and already has a sane concurrency model, but the lack of libraries and community interest seems to make that pretty much a non-starter.
- DougN7 9y agoWell, that and Perl has to be one of the most unreadable languages out there ;)
- vgy7ujm 9y agoOnly if you don't know Perl... To me Python is more unreadable ;)
- bsder 9y agoOnly if you know ALL of Perl. I used to carry around a Perl program of my own on a printout to take to VLSI interviews. That way when I got the "Do you know Perl?" question I could bring it out and force the interviewer into MY stupid subset of Perl rather than being stuck in his stupid subset of Perl. That's not a compliment to the language.
- vgy7ujm 9y agoIs Python really any different or is it just wishful thinking? Why do I see Python programs that look like a weird mix of Lisp and Java? Surely it is because even with what Python enforce, there are many many ways to produce unclear code that really don't even has to do with the language used. And why did my employer see the need for the comprehensive Python style guidelines manual... I guess Python bit just as hard as Perl. Also now that Python is used more by newbies and non-programmers that is where more of the bad code end up (aka Perl late 90s, still pollutes the internet). The quality of Perl frameworks, libs, example code etc is actually increasing and getting easier to find.
- est 9y agoSub-interpreter looks like an interesting idea, I don't mind limited to few primitive immutable objects shared between threads as long as it's shared.
- wiremine 9y ago> We estimate a total cost of $50k... Just looking at it from a financial perspective, having a great Python interpreter that doesn't have a GIL seems like a no brainer for $50,000, and it creates another reason why people should take a look at PyPy. Side note: if you haven't looked at PyPy, check it out, along with RPython https://rpython.readthedocs.io/en/latest/ https://rpython.readthedocs.io/en/latest/
- andruby 9y agoHow can they estimate this? What about all the libraries that might not be compatible with the solution PyPy comes up with? This feels like a number that might in the end blow up to 10x the original estimate.
- joaodlf 9y agoThis reminds me how the sales/marketing teams in my company typically sell new features: "Not having this feature costs us 50k a month!" It might as well not be the case here, I just found it funny, 50k is our little magic number.
- rguillebert 9y agoIt's not the PyPy developers' job to make every Python library threadsafe, people writing libraries will have to make their code threadsafe, like in every other language.
- taeric 9y agoThere is a clear difference here, though. Making a change that could lead to poorly written libraries now being broken is clearly the fault of the change. Userspace for these libraries is defined by how it is, not how it was intended. (And really, was it intended to be dangerous in this way?)
- masklinn 9y ago> There is a clear difference here, though. Making a change that could lead to poorly written libraries now being broken is clearly the fault of the change. No, these libraries are already semantically broken in the same way e.g. libraries which didn't properly close their files and assumed the CPython refcounting GC would wipe there asses were broken. They're already broken under two non-GIL'd implementations.
- sandGorgon 9y agoThis is a PERFECT usecase for Kickstarter. It makes me sad that this is a blog post that made it number 1 on HN with vast readership with open pursestrings.. yet there is not a campaign fundraising link. Use Kickstarter or Plasso to sell a pypy pro license - its so much easier for companies to pay invoices than to donate. If nothing else, I would pay for an official conda pypy package which works seamlessly with pandas and blas.
- brian_herman 9y agoThey can raise funds on their own for example: $67126 of $105000 (63.9%) for py3k in pypy And for STM in pypy: 2nd call: $59080 of $80000 (73.9%)
- fijal 9y agoMaybe selling packaging is a good idea.... that said, kickstarter does not work in any jurisdiction we can use.
- dmix 9y ago> yet there is not a campaign fundraising link. Did you read the article? They said in the article they aren't asking for individual donations at the moment: >> we would like to judge the interest of the community and the commercial partners to make it happen (we are not looking for individual donations at this point) Plus I'm sure they will consider using Kickstarter when the time comes.
- dweekly 9y agoCash is one objective way to discern interest.
- notzorbo3 9y agoI think they're looking more for corporate backers. They're probably very aware that their GILless PyPy will not run many of the programs and libraries out there that are not written to be thread-safe. And when the GIL is in place, there's really not much reason to write thread-safe code. At the very least you won't notice much when you're writing code unsafe code. So I assume they're not doing a kickstarter to prevent the following from happening: 1. The internet at large will assume they're going to get a GILless PyPy that can actually run their code. 2. A separate PyPy is released that doesn't run their code. 3. People are angry that they didn't get what the thought they were gonna get, like what often happens with kickstarter backed projects. 4. With no coporate support and waning public interest due to the uselessness of a GILless PyPy, the separately released project becomes unmaintained.
- crb002 9y agoHaving ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.
- kevingadd 9y agoThe post addresses this strategy and describes why they consider it insufficient. You can already do this in Python, anyway.
- sametmax 9y agoYes but they discard multiple interpreters as having no real advantages. This is dishonest since it should be able to share objects with much, much less overhead as multiprocessing, while allowing to use multiple CPU. It's not perfect, but honestly it seems like a very good deal for Python.
- metalliqaz 9y agoI agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.
- pslam 9y agoAbsolutely agree. Almost all tasks will perform very well when using multiprocessing. It also has a nice side-effect of steering you towards explicitly coding data flows without fine-grained sharing. If you need to close that gap between the performance of multiprocessing, and multithreading, then you probably shouldn't be using Python, or any language of the same shape, in the first place. There is one other option I'd like to see: multiprocessing style, but with multiple Python interpreter instances in the same process — one per thread. There would still be the hard delineation of data boundaries between instances, but less overhead for pushing data between them.
- twoodfin 9y ago"It mostly works for simple programs, but probably segfaults on anything complicated" is not a promising beginning. Starting with race condition chaos and trying to patch your way out of it with "strategic" locking a) Inspires much less confidence than starting with a known-correct locking model (the degenerate case being a GIL) and preserving it while improving available concurrency. and b) Seems at least 50/50 to end up without much in the way of tangible scalability gains once enough locking has been added to reduce the rate of crashes and data corruption to an acceptable (?!) degree. At least that was my takeaway from all the challenges Larry Hastings has documented while working on the gilectomy. Sure, they don't have to worry about locking around reference counting, but it's not like writing a (concurrent?) GC operating against concurrently executing threads isn't a significant design challenge itself with many tradeoffs to make.
- mcherm 9y ago> "It mostly works for simple programs, but probably segfaults on anything complicated" is not a promising beginning. Perhaps they would have done better to say "it works correctly for all programs that do not assume the built-in data structures are threadsafe". That is an accurate description, what you quoted is a reasonable approximation.
- fijal 9y agoThe concurrent garbage collector has been already written. It probably has bugs, but the basis has been done for the STM effort and redone now. The mess of race conditions is surely not a great place to start, which is why it would take a man year to finish :-) I don't see a much better starting point tbh, but to look at all the mutable data structures in python (of which there are too many) and try hard
- jtchang 9y agoJust reading this post makes me think that it could do with a bit more "marketing" speak. I love python. I use it day to day and realize there is a GIL. But give me some business reasons as to why removing the GIL is critical. Will is save me a ton of money? Will my stack magically just run faster? I wonder if Google has already done so since they would benefit quite a bit from a GIL-less python.
- deleted 9y ago[deleted]
- andreasgonewild 9y agoI just can't stop thinking that somewhere along the line one of the Guidos should have reacted to handing out global locks left and right. I mean, as long as its only you and your friends using it. But once it starts spreading, these are the kind of issues that need to be kicked out of the way asap. Lock granularity affects the entire design of client code, reducing it basically means rewriting everything. Ah well, at least it serves as a warning sign for budding language composers as myself. Snabel did full threading before it walked or talked: https://github.com/andreas-gone-wild/snackis/blob/master/snabel.md https://github.com/andreas-gone-wild/snackis/blob/master/sna... And to any Pythoneers with sore toes out there: pick a better language or learn to live with it, down-voting me will do nothing to solve your problems. It's a tool, we're supposed to pick the best one for the job; not decide on one for life and defend it to death. Imagine what could happen if language communities started working together rather than competing. There is no price to be won, we're all being taken for a ride.
- deleted 9y ago[deleted]
- vladf 9y agoThere seem to be a lot of naysayers in the comments about removing the GIL. Multiprocess parallelism isn't always appropriate, so I find this to be a very promising change that will definitely make me want to switch to PyPy. Here are the use cases I've found multiprocessing to be inappropriate: * High-contention parallel operations. Doing synchronization through a Manager (a separate IPC-based synchronizing broker process) is of course less preferable than, say, a futex. * Embarrassingly parallel small tasks. This is a big one. If the operation being parallelized is short, then message-passing overhead takes up more runtime than the operation itself, like a bad Amdahl's Law scenario. Shared address space multithreading solves this problem. * Related: parallelization without the pickling headaches! Many objects can be synchronized but not easily pickled or copied! True multithreading would really enable a large amount of use cases (map a lambda instead of a named function, anyone?) since the same Python interpreter can just pass a pointer to a single shared object. * Related: lots of libraries (Keras, TensorFlow, for instance) make heavy use of module level globals, and aren't meant to be run on multiple cores on the same machine (TF, for instance, hogs all GPU memory). Multithreading in these deep learning environments (assuming PyPy support from those packages) is useful for parallelizing the input ingestion pipeline. But this point isn't TF/Keras dependent; I can't recall other modules but don't doubt the heavy use of module-globals that's unfriendly with fork()-ing, especially if kernel-related state is involved.
- njharman 9y ago> Multiprocess parallelism isn't always appropriate Using Python isn't always appropriate.
- gshulegaard 9y agoI feel like the GIL is, at this point, Python's most infamous attribute. For a long time I thought it was also the biggest flaw with Python...but over time I care less and less about it. I think the first thing to realize is that single-threaded performance is often significantly better with the GIL than without it. I think Larry Hasting's first Gilectomy talk was extremely insightful (about the GIL in general and about performance when removing the GIL): https://youtu.be/P3AyI_u66Bw?t=23m52s https://youtu.be/P3AyI_u66Bw?t=23m52s I am not sure I would, personally, trade single-threaded performance for enabling multi-threaded applications. I view Python as a high-level rapid prototyping language that is well suited for business logic and glue code. And for that type of workload I would value single-threaded performance over support for multi-threading. Even now, a year later, the Gilectomy project is still slightly off performance-wise (although it looks really really close :) ): https://youtu.be/pLqv11ScGsQ?t=27m32s https://youtu.be/pLqv11ScGsQ?t=27m32s As noted elsewhere, multi-processing offers adequate parallelization for this type of logic. Also, coroutines and async libraries such as gevent and asyncio offer easily approachable event loops for maximizing single-threaded resource utilization. It's true that multi-processing is not a replacement for multi-threading. There definitely are tasks and workloads where multi-processing and its inherent overhead make it unsuitable as a solution. But for those tasks, I question whether or not Python itself (as an interpreted, dynamically typed language) is suitable. But that's just my $0.02. If there is a way to remove the GIL without negatively impacting single-threaded performance or sacrificing reference counting for a more robust (and heavy) GC, then I am all for it. But if there is not...I would just as soon keep the GIL.
- xenadu02 9y agoMany projects have solved this problem with dual compilation modes and provide two binaries the user can select from at runtime. Eliminating the GIL doesn't have to mean actually eliminating it. You could certainly have #defines and/or alternate implementations that make the fine-grained locks no-ops when compiling in GIL mode. Conversely make the GIL a no-op in multithreaded mode.
- VectorLock 9y agoThe GIL has been a much bigger problem for perception than it ever has been for performance. Python has lost more mindshare over it than anything else. The few machine cycles that were ever saved by moving away from it were far outweighed by the waste of human cycles.
- eslaught 9y agoThe comments here are missing a massive use case: shared memory. Shared memory isn't just about programmer convenience. It's about using a machine's memory resources more effectively. Yes, shared memory is available in multi-processing, but it doesn't necessarily interact well with existing codes. I've been working on adding Python support to Legion [1], a task-based runtime system for HPC. Legion wants to manage shared memory so that multiple cores don't necessarily need multiple copies of the data, when the executing tasks don't conflict (all are read-only, or access disjoint data). Legion is C++, so this mostly "just works". Some additional work is required to support GPUs, but it's still not so difficult. But with Python, if we go with multiprocessing, we have to switch to a different mechanism. Worse, Python is an optional dependency for Legion, so we can't depend on Python's multiprocessing support either. If you have a large existing project, and a use case that can take advantage of shared memory, being forced into Python's multiprocessing scheme for parallelism is a pain. We've been investigating using a dlmopen approach as well, based on this proof of concept [2]. Turns out that dlmopen in every available version of libc has a critical bug that prevents it from being practically useful, if you have any desire to make use of native modules. You can build a custom libc with this patch [3] but rolling a custom libc is also a massive pain. In all likelihood we'll end up rolling our own multiprocessing to make this work. If the GIL were truly gone though, we could potentially avoid many of these issues. [1]: http://legion.stanford.edu/ http://legion.stanford.edu/ [2]: https://news.ycombinator.com/item?id=11844268 https://news.ycombinator.com/item?id=11844268 [3]: https://patchwork.ozlabs.org/patch/496559/ https://patchwork.ozlabs.org/patch/496559/
- sametmax 9y agoBut then multi-interpreters would allow that, and the article discard it as a valid solution. I find it harsh. It seems much easier to implement, doesn't have the same serialization problem than multiprocessing has and allow to utilize all the CPUs. Yes it's not as good proper threads because you do have more overhead, but it's an order of magnitude better than what we currently have, while being way easier to do that getting rid of the GIL. Too bad the current project is on hold.
- sitkack 9y ago
- sobkas 9y ago>fully working PyPy interpreter with no GIL as a release, possibly separate from the default PyPy release I have concerns that if such functionality will not be in the main release enabled by default(and consequently don't get as much testing), it will just bitrot and in the end, will be removed.
- _wmd 9y agoThey're asking for funding to spend on a risk-free attempt at GIL removal (risk-free since it won't bone PyPy mainline), if the attempt meaningfully succeeded I'd imagine their next step would be making it the default. A fully functional PyPy that could do heavy math in multiple threads would be an amazing tool in the box, but there are plenty of risks to that (penalizing single threaded performance, for example). So this strategy makes plenty of sense to me. They can't just do it on mainline from the outset because there are huge obstacles to overcome.. for example, that ancient foe, CPython extension interface compatibility, which assumes a single global lock covering all mutable extension data. I don't think there will ever be a way around maintaining the GIL for that, even if pure Python code can freewheel it otherwise
- unkown-unknowns 9y ago> If we can get a $100k contract, we will deliver a fully working PyPy interpreter with no GIL as a release, possibly separate from the default PyPy release. If done as a separate release, will that version be maintained in the future?
- smegel 9y agoRemoving it is the easy part.
- cool-RR 9y agoPython's GIL issue is like the Israeli-Palestinian conflict. 1. People like to talk about it a lot, complain about it and say their opinion of what should be done with it. 2. It's not likely to be resolved for years to come. 3. In the end, the problem has very little effect on people's lives, much much less than the amount of hype around the issue.
- ericfrederich 9y agoPerhaps a little unrelated, I used the rpyc package to get Jython and CPython working together. In the end I was able to use Java libraries from CPython pretty much seamlessly.
- vram22 9y agoYou mean you used RPyC at both ends, on the CPython side and on the Jython side. Cool idea. I knew about RPyC but had not thought of using it in this way. And getting access to Java libraries by doing this, can be very useful, I can see.
- issaria 9y agoIt's not going to happen, you not only have to fix all the legacy code, but also fix the developers.
- tonybaker 9y agoGET RESULTS IN ONE HOUR TIME FOR ALL KIND OF EMAIL HACK.Change School Grades? Hack Banks? Erase Criminal Records? Hack Websites? Hack Database? Hack Drivers license? Hack Call Log? Hack Visichat and Flashchat Rooms? Hack FTP User and Pass? Hack Facebook,Whatssap,Twitter,Instagram,Webcam etc.Hack VB Forum? Hack Wordpress Blog? Hack CC any Country? Hack Money Booker Account? Hack Liberty Reverse Account? Hack Paypal Account? Root Server? By Pass Google Phone Verification? Install Red on Linux Server? Hash Crack? DDOS Server? Retrieval of Lost Files and Documents CONTACT: empiricalhackers4187@gmail.com
- Pxtl 9y agoIck, I'd forgotten how monkeypatchable the core of Python was. If not for that, I'd focus on supporting some kind of pseudo-process where multiple instances of the Python interpreter could be loaded but they would only share pure-functional libs which, I assume, could be used in a threadsafe fashion... but then you run into the mutability of those libs. Well, the mutability of everything in python. Plus what happens if those libs expose anythign that you could hold a reference to - what happens to refcounting in a multithreaded Python? Honestly, I feel like the world has passed Python by. At this point the cost of its performance limitations don't seem to be worth its payoff. Not that it's a bad language - I like Python. I just don't really feel the need to use it for anything anymore.
- stuaxo 9y agoI liked "STM" as a big-idea approach, but can see how going the traditional way may bear fruit more quickly. Could the experience gained this way (and by other projects such as the gilectemy) help with a future STM attempt ? I wonder if we need better hardware for STM to work well too.
- cjbillington 9y agoThis seems like a good place to spruik something I made, a Python package for profiling how much the GIL is held: https://github.com/chrisjbillington/gil_load https://github.com/chrisjbillington/gil_load In my experience, the GIL is not held for nearly as high a proportion of the time as people think it is, because properly written C extensions and blocking io always releases the GIL. So long as the proportion of time the GIL is held is not approaching 100%, then you can still get gains from threading. This is almost always the case in numerically heavy code that uses numpy or scipy, since the extensions release the GIL. Threads work almost just as well at speeding up this code as in any GIL-free interpreter. And usually long before you consider multithreaded code, you'll want to move the bottlenecks of your code over into Cython or something, since that can give speedup factors much larger than multithreading. In which case all you need is a "with nogil:" around the the meaty bit of you Cython code, and then it too will be able to get speedups from multithreading.
- ComputerGuru 9y agoI can't believe it's 2017 and the official pypy updates come from blogspot; I thought this was a plea from a community member. Anyway, really good on them to finally move on killing the GIL. It's been a long-time issue - the type that only gets worse the longer you ignore it. That said, I think today Python and GIL are synonymous and the entire Python ecosystem has almost evolved around the GIL. While I'm sure there are applications that would benefit from its removal, I think in the whole, the ecosystem will not change much because of this.
- denfromufa 9y agoWill this work with CFFI?
- andy_ppp 9y agoGreat, and I want be a billionaire with washboard abs. The main problem is none of the Python code that currently exists is thread safe so you might as well start again from scratch. Python is a needlessly complicated language with two important things; Numpy and TensorFlow. These use Python as a scripting language for C. Just move to Go, Scala or Elixir/Erlang if you want to avoid the GIL (or write anything parallel). You can thank me later!
- memracom 9y agoPersonally, I don't think the GIL matters. First of all most of us run apps on Linux which has reduced the overhead of processes so much that threads have lost much of their advantage. Secondly, people understand that locks are generally a bad thing to use unless you really are a threading/locking rocket scientist. Most mere mortal developers are better to use message queues. Even the Java world has mostly given up locks in favor of java.util.concurrent which was implemented by serious experts to handle all of the corner cases that you would not think of. Third, using an external Message Queuing system like RabbitMQ gives you other benefits. And fourth, writing distributed apps glued together by message queues helps you avoid the dreaded Big Ball of Mud. At this stage in Python's evolution, I view the GIL removal as a computer science project that some people will implement again, and again, just to learn or to exercise their chops. Great idea! Just don't demand that the entire community of Python developers goes down your road. If CPython never gets rid of the GIL that suits me just fine. GIL free programming can be done on other implementations of Python like Jython and IronPython. As far as PyPy is concerned, as long as it does not disrupt the use of PyPy as a means of speeding up a CPython app from time to time, then have fun.
- nurettin 9y agoComing from mobile and desktop programming, most use cases I've seen for threads revolve around doing something in the background to keep user interfaces responsive. That use case already has a threadsafe queue. The UI queue. When your thread finishes or is ready to signal progress, you queue the event to the UI thread and forget about it. Now I've been following this pattern for a long time and have no a experience dealing with GIL How this removal of GIL going to effect this use case if at all?
- fijal 9y agoHi, blog post author here. Let me put an offer here: If you want to ask a question that warrants a response (as opposed to promoting your own effort, which is valid but does not warrant a response), please mail me, the mail is public and I'll put the responses publically on either my blog or pypy blog.
- alfanerd 9y agoI just made a quick test: CPython-3.6.1 vs. Jython-2.7.0 (May 2015) I ran Larry Hastings' Gilectomy testprogram x.py: fib(40) on 8 threads HW: MacBook Pro, 2015, 8 (4+4) cores, 1Gb RAM Jython ran the program 8 times faster, utilising all 8 cores >95%. Python ran on 1-2 cores less than 60% utilisation. (Pretty sure Jython will run 16 times faster on 16 cores) It's 2017, why this is acceptable to GvR and the Python community is beyond me. Jython: real 1m4.959s user 7m38.521s sys 0m2.396s Python: real 8m19.035s user 8m16.508s sys 0m11.424s
- zepolen 9y agoCould you post the code for that fib program I couldn't find it anywhere.
- alfanerd 9y agoIt is in the Gilectomy branch of Larry Hastings github project. (https://github.com/larryhastings/gilectomy https://github.com/larryhastings/gilectomy) I also pasted the fib test to pastebin: https://pastebin.com/Ryyb2K7V https://pastebin.com/Ryyb2K7V
- zepolen 9y agoAh so just the naive recursive fibonacci on 8 threads with no data sharing between them. Interestingly doing the same on Cpython using the multiprocessing module was ~2x slower than jython/threads. More interestingly pypy with multiprocessing was ~5x faster than jython/threads. $ time jython fib.py 40 real 1m11.247s user 6m14.130s sys 0m3.012s $ time python fib.py 40 real 2m4.067s user 11m46.103s sys 0m2.352s $ time pypy fib.py 40 real 0m21.040s user 1m51.461s sys 0m1.892s
- faragon 9y agoDoes it work also in ARM architecture ("weak" memory model), or just in x86/x86-64-like ("strong" memory model)?