23 ms·
Removing the GIL will not break the vast majority of existing Python code. Here's the backwards compatibility section of PEP 703: https://peps.python.org/pep-0
by js2 3y ago
Removing the GIL will not break the vast majority of existing Python code. Here's the backwards compatibility section of PEP 703:
https://peps.python.org/pep-0703/#backwards-compatibility https://peps.python.org/pep-0703/#backwards-compatibility
That section is short and I encourage you to read it in full. Here's what it says about how removing the GIL affects existing Python code:
• Destructors and weak reference callbacks for code objects and top-level function objects are delayed until the next cyclic garbage collection due to the use of deferred reference counting.
• Destructors for some objects accessed by multiple threads may be delayed slightly due to biased reference counting. This is rare: most objects, even those accessed by multiple threads, are destroyed immediately as soon as their reference counts are zero. Two places in the Python standard library tests required gc.collect() calls to continue to pass.
That's it with respect to Python code.
Removing the GIL will require a new ABI, so existing C-API extensions will minimally need to be rebuilt and may also require other changes. Updating C-API extensions is where the majority of work will be if the PEP is accepted.
This will be an opt-in feature by building the interpreter using `--disable-gil`. The PEP is currently targeted at Python 3.13.
This is nothing like the Python 2 > 3 transition.
- Waterluvian 3y agowhich python which python3 which python3nogil
- echelon 3y agovenv, virtualenv, pipenv, pyvenv, venvwrapper, conda, ... Python4 needs a hard reset.
- whalesalad 3y agothose are all means to the same end. venv == virtualenv virtualenvwrapper is ancient not rly used anymore pyenv is a third party tool that makes some of this easier, notably around creating more than just a virtual env in that you also choose the Python version. Python is not hard to deal with in this regard I think people are just uninformed.
- coldtea 3y agoAre you kidding me? The horrendous way Python does dependency management and virtual environments, and the fractured ecosystem around those, is one of it biggest pain points, often covered by core CPython developers and prominent Python third party developers, hardly "misinformed" people. https://xkcd.com/1987/ https://xkcd.com/1987/
- whalesalad 3y agoThat comic is very old. In the days of 2.x it was a little harrier but nothing like people make it out to be. The literal only thing you need to understand is “sys.path”. If you inspect this in a shell you will know what you’re up against. Python is all literally just directories. It’s so easy and yet people get so bent out of shape over it. Create a venv, activate it, and use pip as normal. If you ever run into issues, look at sys.path. That’s it.
- coldtea 3y ago>That comic is very old. And few things have changed since then.
- whalesalad 3y agoI’m gonna have to disagree with ya there bud. Never been easier to be a Python developer.
- coldtea 3y agoWhich is irrelevant. We're talking about the dependencies/packaging/virtual environments situation, not whether "it's easy to be a Python developer" in general. And you can disagree all you want, but it's simply wrong that Python's packaging/venv ecosystem is "just fine".
- beowulfey 3y agoThere are options to do things other ways, but most of the time I just use venvs and pip for everything. Is it because people have to use venvs that people complain about it? I’ll admit being able to install via an OS package manager, vs pip, vs anaconda etc etc can be confusing, but is any of that really Python (the language)’s fault?
- replygirl 3y agomore like which python3-compat
- bick_nyers 3y agosudo apt install python3nogilispython3 sudo apt install python3ispython
- tremon 3y agoYou would not need separate packages to do that (in fact, you can't do this with separate packages because dpkg will complain if two packages provide the same file). sudo update-alternatives --set python3 /usr/bin/python3-nogil sudo update-alternatives --set python /usr/bin/python3
- takeda 3y agoThe PEP actually states that the nogil version would also have env variable allowing to temporarily enable GIL. Although I guess in practice they might still build separate versions.
- js2 3y agoWhat do you propose as an alternative? I write code in a lot of languages and I can't think of a single one where I don't have to consider the version. This applies to C, node, ruby, swift, gradle/groovy and java at least. Even bash. When developing for Android and iOS, I have to consider API versions.
- echelon 3y agoAlmost every third party ML model I look at seems to have different versions, different dependencies, and requires deliberate trial and error when creating container images. It's a mess. Having interpreters and packages strewn across the machine is a nightmare. The lack of standard tooling has created a lawlessly dangerous wild west. There are no maps, no guardrails, and you have to beware of the hidden snakes. It goes against the zen of python. As a counter example, Rust packs everything in hermetically from the start. Python4 [1] could use this as inspiration. Cargo is what package and version management should be, and other languages should adopt its lessons. [1] Let's make a clean break from Python3 even if we don't need a new version right now.
- Waterluvian 3y agoRust and Node (via nvm) feel good. The worst I run into is “this version of node isn’t installed” and then I just add it. And I don’t have to worry about where dependencies are being found. Python likes to grab them from all over my OS.
- js2 3y agoI use direnv and pyenv. When I cd to a repo/directory, the .envrc selects the correct Python and the directory has its own virtual environment into which I install any dependencies. I don't find that Python grabs packages from all over the OS.
- Waterluvian 3y agoIndeed there’s a good half-dozen options in Python for this kind of thing. But it depends on which one each project opts to use.
- Nihilartikel 3y agoAsdf makes all of this pretty easy. For consulting I often need multiple versions of everything to match client projects - Just install them with asdf and put a .toolversions file in the project folder with the desired tooling builds.
- EamonnMR 3y agoIt's probably going to be /home/you/project/venv/bin/python
- Der_Einzige 3y agoJust use conda/mamba unironically. Solves all of these problems. Venv and other solutions should be depreciated already.
- kaba0 3y agoIt is not the language’s task to solve. Use a proper dependency manager like nix.
- meowface 3y agoOr pyenv (https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv) if you don't want to take the plunge into something like nix. As for managing Python library dependencies, I use poetry (https://python-poetry.org https://python-poetry.org), though unfortunately both it and pipenv seem to progressively break functionality over time for some reason.
- takeda 3y agoIt would be more like: does the code support nogil? Because the code that is updated is expected to work with both versions.
- bsdz 3y agoIt's worth highlighting that the GIL will still be available even when compiled with --disable-gil. > The --disable-gil builds of CPython will still support optionally running with the GIL enabled at runtime (see PYTHONGIL Environment Variable and Py_mod_gil Slot).
- packetlost 3y agoWhy not do it the other way? Have an opt-in at runtime for no-gil?
- radlad 3y agoBecause in general, removing the GIL should be an improvement.
- gcbirzan 3y agoIn general, until you do use threads, removing the GIL is a performance loss. Even if you do use threads, you have an overhead: https://peps.python.org/pep-0703/#performance https://peps.python.org/pep-0703/#performance
- packetlost 3y agoOh, that makes sense! I forgot that there would be overhead related to the changes for no GIL.
- ilyt 3y ago> This is nothing like the Python 2 > 3 transition. Well what py2to3 promised was also not what happened and by far one of worst disasters of a migration in history of open source so you can't blame people for being skeptical
- leoh 3y ago[flagged]
- croshan 3y agoPosting a Wikipedia link without commentary or context is essentially a text meme. It doesn’t invite discussion. What are you trying to say? (Not that I disagree with your point, but it’s unclear.)
- myhf 3y agohttps://en.wikipedia.org/wiki/Thought-terminating_clich%C3%A9 https://en.wikipedia.org/wiki/Thought-terminating_clich%C3%A...
- leoh 3y ago> Posting a Wikipedia link without commentary or context is essentially a text meme. It doesn’t invite discussion. You literally just replied and we are engaging discussion — empirically the wiki link above has done literally the opposite of what you are saying! At any rate, the intended relevance of the wiki link regarding loss aversion is the following: When something bad happens to someone, they tend to overestimate how bad it is relative to something experienced as objectively equally good; it is thus important to be aware of such a bias — in addition to using regular old critical thinking. In this case, critical thinking could look like the following: “although this is a ‘migration’, what do we mean when we say that? and is it really the same thing as py2 to py3?” I would tentatively propose “no” to the latter question; and would additionally propose that calling this a “migration” is not terribly useful as although it’s not incorrect, it’s insufficiently specific and seems to invite a category error. In addition to the loss aversion bias mentioned above. Category error: https://en.wikipedia.org/wiki/Category_mistake https://en.wikipedia.org/wiki/Category_mistake
- d1l 3y agoYou've conveniently described python behavior but lots of C code relies on the gil implicitly and will need to add locking to be correct in a nogil world. I'm not saying for sure this is bad! I do think it is dishonest though about the potential impact. Lots of critical libraries are written in C.
- js2 3y agoDishonest? I didn't hide that fact: "Removing the GIL will require a new ABI, so existing C-API extensions will minimally need to be rebuilt and may also require other changes. Updating C-API extensions is where the majority of work will be if the PEP is accepted."
- eikenberry 3y agoBut it was downplayed in the comment when it was always the #1 reason for keeping the GIL. Python level changes were never a serious part of the argument.
- electroly 3y agoYeah but everyone knows that is obviously the biggest issue. Nobody was really concerned about the Python code which is the part that you said will go swimmingly—that's table stakes for a change at all. We already assumed the pure Python code would upgrade easily; a change would have no chance whatsoever of being accepted otherwise. Everyone is worried exclusively about the C extensions, and always has been. This seems to have been presented as a new approach to GIL removal that fixes the problem with C extension breakage but it's just the exact same old approach we've always been considering that breaks the C extensions. No-GIL ain't done until all the C extensions run, especially NumPy.
- js2 3y ago> Nobody was really concerned about the Python code This was the top comment when I added my original comment: "Fundamentally every single piece of python code ever written will have to stop and now worry about potential race conditions." https://news.ycombinator.com/item?id=36644114 https://news.ycombinator.com/item?id=36644114 Most of the other comments here at the time were similarly from people who clearly hadn't read the PEP saying it would break all existing Python code. So I did my best to represent the backwards compatibility section of the PEP. I told folks to go read it and linked to it. I cited the portion relevant to Python code. There were too many bullet points for the C-API, so I summarized it with the disclaimer "Updating C-API extensions is where the majority of work will be if the PEP is accepted." I also read the PEP discussion thread where there was disagreement from Python maintainers on how much work would be required of C extension authors, but most of the folks stating it would be a lot of work hadn't seemed to actually have tried to port anything to the new API. Meanwhile Sam had asserted that: > Most C API extensions don’t require any changes, and for those that do require changes, the changes are small. For example, for “nogil” Python I’m providing binary wheels for ~35 extensions that are slow or difficult to build from source and only about seven projects required code changes (PyTorch, pybind11, Cython, numpy, scikit-learn, viztracer, pyo3). For four of those projects, the code changes have already been contributed and adopted upstream. For comparison, many of those same projects also frequently require changes to support minor releases of CPython. I don't think I've misrepresented anything. > No-GIL ain't done until all the C extensions run, especially NumPy. Sam already got NumPy working without the gil: https://youtu.be/9OOJcTp8dqE?t=1958 https://youtu.be/9OOJcTp8dqE?t=1958
- bilsbie 3y agoCould it be opt in at run time though?
- ameliaquining 3y agoI think the hope is to get there eventually, but since it's a big change, they're doing it one step at a time.
- rogerbinns 3y agoI'm the author of a Python C extension, and having the GIL gone will be a lot of work. Code currently looks like this: * C function called with GIL held * Extract data needed to do work * Release GIL * Do work * Reacquire GIL * Modify data, build result * Return As an example of the changes, a list could be passed in. I would need some form of locking while processing that list so that mutations while processing won't crash the code. The GIL does currently result in robust code by default because data can't mutate underneath you. Without the GIL the code will appear to work, but it will be trivial for an attacker to use mutations to crash the code. Expect huge numbers of CVEs.
- pierat 3y agoC'mon folks! MOD THIS UP This is the lead author of the majorly potential impacted extension!
- gjulianm 3y agoAs far as I know PEP703 includes provisions to make operations on containers thread-safe. That should at least avoid most crashes. Borrowed references can be more problematic, but it seems most cases could be fixed by replacing GetItem with FetchItem calls. Overall, as another C Python extension, I don’t really think it’s going to be that much of a pain. In fact, I could even get away with no changes (other than build fixes and such) if, for example, I guarantee that each “main object instance” is only accessed by one thread, and I’d still get a lot of benefits from the nogil.
- bjourne 3y agoHere is code you see all the time in extensions: if (PyList_Size(lst) >= 3) { PyList_SetItem(lst, 0, ...); PyList_SetItem(lst, 1, ...); PyList_SetItem(lst, 2, ...); } Whether the individual operations are thread-safe is irrelevant. For the code to be thread-safe it needs to acquire lst's lock before the if statement and release it afterwards.
- deleted 3y ago[deleted]
- zzzeek 3y ago> Destructors and weak reference callbacks for code objects and top-level function objects are delayed until the next cyclic garbage collection due to the use of deferred reference counting. this actually does "break" a lot of things, as you would be surprised how much code implicitly relies upon cPython's behavior of immediately calling weakref callbacks when an object is dereferenced. This is why keeping test suites running on pypy can be difficult, because it has the latter behavior. > Removing the GIL will require a new ABI, so existing C-API extensions will minimally need to be rebuilt and may also require other changes. Updating C-API extensions is where the majority of work will be if the PEP is accepted. as you note, there will be *two* versions of the Python interpreter. that means every C extension has to be built *twice*. against both versions of the interpreter. Go look at how many files one must have available when publishing binary wheels: https://pypi.org/project/SQLAlchemy/#files https://pypi.org/project/SQLAlchemy/#files the number of files for py3.13 now doubles. not clear if we actually have to have both Python builds present, so I would have like /opt/python3.13.0.gil and /opt/python3.13.0.nogil ? if the gil removal changes almost nothing, why have two versions of Python?
- imtringued 3y agoI don't see why you couldn't use updated "No GIL" extensions with the GIL interpreter. The "No GIL" interpreter mode will simply abort if you load an unsupported C extension.
- omnicognate 3y ago> existing C-API extensions will minimally need to be rebuilt and may also require other changes. This vastly understates the work involved. Most C extensions and embeddings will require major structural changes or even rewrites. These things are everywhere and are a major reason for python's popularity. A typical financial instition, for example, will have a whole bunch of them, with a morass of python code built on top. Many of these companies took ages to transition to python 3 (and many still have pockets that haven't!). For them to remove GIL reliance from their C extensions is a much bigger ask than migrating to 3 was, and their response to being asked to do it is likely to be blunt. The fact that only a small minority of developers involved in python directly use the C API (and fewer understand the consequences of GIL removal for them) means the issue tends to be overlooked in discussions like this, but it's the reason a GILectomy will be worse than 2 to 3. In practice it's likely to end up being either a fork or a mode you have to switch on/off, both of which would be miserable for everyone. The GIL is part of python's success story. It made it easy to write extensions, and the extension ecosystem made the language popular. Every language doesn't have to converge to the same endpoint. Different tools are suited to different jobs. Let it be.
- deleted 3y ago[deleted]
- floomk 3y agoIt would be helpful and interesting if you could explain why the impact is so huge
- gary_0 3y agoThey're talking about C code that blindly relies on implicit concurrency guarantees that are now conditional. Isn't that enough said?
- mrlyx 3y agoNot at all. The GIL was explicitly part of the C-API! No one used the guarantees accidentally, including CPython itself. The GIL was one of the reasons why the C module ecosytem took off.
- riazrizvi 3y agoI think the primary problem with removing the GIL is the performance from degraded garbage collection. To quote the issue with an implication that it isn’t a big deal because it’s not many words, doesn’t make the case for me. The delay in when garbage collection happens is the only reason to my mind why Java sucks balls as a memory hog vs C-based languages. Process performance is in large part based on when critical lowest latency memory resources can be freed for the top level working object set.
- OOPMan 3y agoUhuh
- ernst_klim 3y agoIt's much worse than 2→3 transition as it seems to me. Python is a very slow language whose main advantage is relatively cheap and easy FFI with bindings to all imaginable library already written. With breaking FFI you'll lose all the libraries while adding SMP to still slow language. It's a cleare loss of value.