6 ms·

The GIL problem won't be solved by throwing engineer hours at it (not even Meta engineers). Fundamentally every single piece of python code ever written will h

• wuiheerfoj 3y ago
  Is this not exactly what the PEP-703 is?
• mountainreason 3y ago
  I dont get this. Just because you don't have the GIL doesnt mean that your previously single threaded code is now multithreaded and stepping on itself.
  • 411111111111111 3y ago
    From previous discussions it's my understanding the c integration is going to be the cause for the issues. From a python perspective it wouldn't necessarily be a big change, but everything can branch out to c, and there you're going to get in trouble with shared memory.
    • dwaite 3y ago
      The most significant impact of this change is that python threads can run at the same time. Before, calls to C API were essentially cooperative multithreading yield points - even if you have 100 python threads, they can't run concurrently under the GIL. 99 are blocked waiting on the GIL. C extensions have always been able to use real threading. But now there is no GIL to synchronize with the python interpreter on ingress/egress from the extension.
  • deleted 3y ago
    [deleted]
  • T-A 3y ago
    No, but it means that your previous multithreaded code is no longer automatically prevented from stepping all over itself by having multiple threads accessing the same data: https://realpython.com/python-gil/#what-problem-did-the-gil-solve-for-python https://realpython.com/python-gil/#what-problem-did-the-gil-...
    • agf 3y ago
      GIL removal doesn't mean "make all of this lockless" it means "replace the GIL with fine-grained locking". So those problems are still solved for Python code. The three issues are the amount of work it takes to do right, the performance cost for single-threaded code, and the CAPI.
      • tomn 3y ago
        finally. you don't even have to read anything to work this out -- if things like dictionary access were no longer atomic that would imply that threaded code without locks could crash the interpreter, which isn't going to happen.
      • hughesjj 3y ago
        I'm simultaneously scared and enlightened seeing all these comments acting as if the GIL is some magic "makes your code thread/concurrency safe" pancea. I always saw it as a shim/hack to make cpython specifically easier to implement, not something that inherently makes code more thread safe or atomic. It's just more work to do things "the right way" across application boundaries, but from my understanding this PEP is Meta commiting to do that work.
        • kzrdude 3y ago
          Removing the lock creates problems in existing code in practice. This is an ecosystem that has less focus on standards and more on "CPython is the reference implementation".
          • hughesjj 3y ago
• stevefan1999 3y ago
  Then just mark the extensions that is compatible with GIL. And also you will have a switch that disables GIL, controllable with a environmental variable or launch option.
• anyoneamous 3y ago
  Yeah, I'm not a full-time developer but even for my bits of scripting I'd be wary of another big change in Python. Personally I'd much rather see this effort go towards a new language which "feels" like Python but adopts more of the development experience of Go and Rust. From my tinkering it seems like Nim might already be that language, in which case what is needed is investment in its package ecosystem.
  • imtringued 3y ago
    Is this supposed to be some kind of joke? The problem with Python 2 to python 3 is that python 3 was essentially a new python 2 esque language instead of just being a major version bump. If python 4 was no GIL python, then both Python and C would remain unchanged as a language.
• segmondy 3y ago
  It will be optional. You don't have to worry about it, set the option to ON and for those who can worry about it, they will have the option to set it to OFF.
  • usrbinbash 3y ago
    The problem is, code that uses on a lot of 3rd party libraries that throws the ON-switch for nogil, will suddenly depend on all these libraries maintainers having worried about this.
    • seunosewa 3y ago
      We can set it up such that if a module imports modules that don't support nogil, nogil will be automatically disabled for them too. So, library designers will be under pressure to update their libraries to enable support. We could also have code tools that detect patterns that aren't GIL safe and throw out loud warnings
      • zarzavat 3y ago
        Why should library authors be put under pressure because someone else chose the wrong tool for the job, and is now trying to push that externality on to the community? Python is a single-threaded language. That’s part of its DNA. The community has already been through one traumatic transition in recent history and the appetite for another one is low. Library authors should not update their libraries to support multithreading, rather the people who want that should be forced to rewrite their code in a language that is more suitable for the problem they want to solve.
        • rolisz 3y ago
          I disagree that single threadedness is in its DNA. It's an implementation detail of CPython. There are other implementations which don't have a GIL even today. Would removing the GIL be a big change for CPython? Yes. But IMO it's worth it
        • 3y ago
• jeroenhd 3y ago
  Getting rid of the GIL would warrant the release of Python 4.0 for me, except the Python project shouldn't be supporting two different branches for as long as they supported 2.7. I imagine there would need to be some kind of annotation to enable the GIL for a method and all of the code it calls, including libraries, so performant Python can take advantage of the lack of a GIL but old code doesn't break. Then all you need to do to maintain compatibility is to annotate your main() and your code should remain compatible for a while. After all, the referenced PEP explicitly calls for making the GIL optional, not for removing it completely.
  • Valodim 3y ago
    "python 4", such a simple phrase yet sure to strike terror in the heart of many
    • brianwawok 3y ago
      As a Python dev who dealt with 2->3 not really.
      • kelipso 3y ago
        Every community has their masochists.
  • imtringued 3y ago
    >Python project shouldn't be supporting two different branches for as long as they supported 2.7. That wasn't the problem. The problem was not giving people the ability to make their code python 3 compatible while they were still stuck with python 2. The python 3 interpreter should have had a python 2 mode that gives you warnings.
    • jeroenhd 3y ago
      Python 2 and Python 3 were inherently incompatible for various reasons (good and bad ones). There were compatibility wrappers for porting Python 2 code to Python 3 but in some instances that was just never going to happen.
• bratao 3y ago
  That is a misconception. The GIL protects the internal state of Python. It don´t make all Python multi-thread code "safe". The PEP 703 still preserve many actual characteristics such as access and writing to dictionaries.
  • slashdev 3y ago
    Correct. Python multi-threaded code is not magically thread-safe. What it did do is make C functions atomic, like dictionary and list methods, since they're implemented by C functions. The reason is the GIL is released and re-acquired while executing Python code (every 100 ops if memory serves), but isn't released by most C functions, making them called with the global lock held, which makes them atomic. You can still have a thread switch between any two Python op codes. e.g. foo += 1 may or may not be atomic depending on how many op codes it compiles to (IIRC, in CPython, it is not atomic.)
    • erosenbe0 3y ago
      Pretty much. This makes it simpler to wrap C libraries and it is partly why Python became successful back in the day.
    • btown 3y ago
      One of the reasons I love using gevent (https://www.gevent.org https://www.gevent.org) is that it's way of introducing concurrency that does preserve these atomicity guarantees! Broadly speaking, the only time your code will be interrupted by something else in the same process is if some function call yields control back to the event loop - so if your block is entirely made up of code you control that doesn't do any I/O, you can be sure it will run atomically. And you get it for free without needing to rewrite synchronous code into asyncio. This does make me wonder, though, if gevent will survive PEP 703's massive changes to internal Python assumptions. That said, gevent does work on pypy, so there's some history with it being flexible enough to port. Hopefully it won't be left behind, otherwise codebases using gevent will see this as a Python 2 -> 3 level apocalypse!
      • 3y ago
• PossiblyKyle 3y ago
  It is solvable by making the hard decision to move to Python 4 with no backward compatibility. The two core issues imo in Python are the GIL and the environment hell and both simply can’t be solved while still keeping the 3 moniker. We’re in a field of constant workarounds and duct tape because we try pleasing too much
  • rolisz 3y ago
    Moving from 2 to 3 was a long and difficult migration, so 3 to 4 will be similarly difficult
    • fmajid 3y ago
      I’ve been using Python since 1994 and my 2-to-3 migration plan was Go…
      • jug6ernaut 3y ago
        The account of time and money ppl put into building and maintaining systems built with scripting languages has never made sense to me.
    • dev_tty01 3y ago
      Seems a bit conclusory. Doesn't this imply the community is incapable of learning from experience?
  • brazzy 3y ago
    Not going to happen for another 10 years at least. Not after how long and painful the move to Python 3 was.
  • bbkane 3y ago
    Python tried that (version 2 to 3) and both the community and dev team were traumatized by the effects enough they've publicly said it'll never happen again. Some things really are too big to change.
• the8472 3y ago
  > perhaps there will be a flag to selectively operate a python process without the GIL (and leave the onous on you to completely test and validate your code works with potentially new and spooky behaviour). Worked for ruby. The original interpreter MRI has a GIL too. Rubinius and JRuby added multi-threading with limited amounts of pain and people fixed libraries over the years. Sometimes just sprinkling lock blocks around a particular FFI calls or only doing them from a dedicated thread will do the job.
• Jasper_ 3y ago
  foo += 1 was never atomic. It always compiled to something like: LOAD_FAST foo LOAD_CONST 1 BINARY_OP + STORE_FAST foo The GIL only protected you during any of those operations, so you can still switch threads waiting between LOAD_FAST and STORE_FAST and have a race. There are a lot of things to be worried about with the GIL conversion of new race conditions that could happen, but there's already too much misinformation out there about the GIL, let's not spread this one even further.
• aardvark179 3y ago
  Having worked on moving a proprietary language from its own green threaded VM to the JVM, and working on TruffleRuby I’m saddened to see this FUD still being trotted out. The GIL and similar mechanisms do not make your code thread safe. It _might_ save you from a small set of concurrency bugs, but they are fewer than you might think, and mostly it will just make intermittent existing issues that little bit more obvious when you move to real threads. Occasionally we would need to fix something in a core library or add a mutex, but those bugs could often be seen in a stress test with green threads or Ruby’s GVL. My guess is the GIL or smaller mutexes will be needed for C extensions and a few other areas, but it’s also likely that could be moved to an opt in mechanism over time.
• ilyt 3y ago
  > Fundamentally every single piece of python code ever written will have to stop and now worry about potential race conditions with innocent things like accessing a dictionary item or incrementing a value Not really, just make those operations atomic or have automatic locking