4 ms·

This can (and I think will) cause issues for C extensions because many are written without multi-threading in mind. Here is a small example which is unsafe if l

This can (and I think will) cause issues for C extensions because many are written without multi-threading in mind. Here is a small example which is unsafe if lst can be accessed from another thread: https://news.ycombinator.com/item?id=36649769 https://news.ycombinator.com/item?id=36649769 Note that the code may cause a context switch even today if the C code callbacks into Python bytecode (via a __del__ method) and the bytecode is long enough (100 instructions I think). However, that is extremely unlikely and much C extension code is not written with such situations in mind. People using C extensions may also rely on them executing atomically. For example, you could have a thread pool that posts and receives from a numpy array. Would work fine today but break without the GIL.

qbasic_forever 3y ago
Yep there are a ton of issues like that to be found, and unfortunately they will manifest as difficult to find and debug race conditions. This is why the proposal and work is to make non-GIL mode entirely optional and not the default. It just means for the brave few that flip it on and use it, be prepared to spend a huge amount of time finding and fixing subtle race conditions in decades of old python library code. The early adopters are going to be in for a lot of pain, or more likely they'll restrict their use of non-GIL processes to very specialized and dedicated processes that have as few dependencies as possible.
- miraculixx 3y ago
  The intent is to make no GIL the default eventually.
  - n2d4 3y ago
    I don't think this is true. There are fairly strong voices on both sides inside the community, at this time it's pretty uncertain. To quote Guido: >Let’s not blow it this time. If we’re going forward with nogil (and I’m not saying we are, but I can’t exclude it), let’s make sure there is a way to be able to import extensions requiring the GIL in a nogil interpreter without any additional shenanigans https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional-3-12-updates/26503/19 https://discuss.python.org/t/pep-703-making-the-global-inter...
    - Alphaeus 3y ago
      The Steering Council said their intention is to remove the GIL-build in future: > Long-term, we want no-GIL to be the default, and to remove any vestiges of the GIL (without unnecessarily breaking backward compatibility). We don’t want to wait too long with this, because having two common build modes may be a heavy burden on the community (as, for example, it can double test resources and debugging scenarios), but we can’t rush it either. We think it may take as much as five years to get to this stage. https://discuss.python.org/t/a-steering-council-notice-about-pep-703-making-the-global-interpreter-lock-optional-in-cpython/30474 https://discuss.python.org/t/a-steering-council-notice-about...
      - toyg 3y ago
        By the usual laws of software estimates, if they think it will take 5 years, it's going to be more like 10-15.
- hinkley 3y ago
  Does Python have a lot of secondary dependencies? I could see someone pulling in two dependencies, not realize they both use the same unsafe library, and end up having them step all over each other.
  - qbasic_forever 3y ago
    It does, so much so things like virtualenv were introduced so every program can have its own set of dependencies such that they won't clash with other libs on your system. Something like flask or fastapi pull in a lot of secondary dependencies alone.
    - hinkley 3y ago
      Probably doesn't work with FFI though? I'm trying to get a sense of how hypothetical versus very real this problem is.
smcl 3y ago
I think the CPython core devs are very keenly aware of these issues. Otherwise they'd have announced a plan to suddenly rip out the GIL altogether, rather than a phased approach that allows people to opt-in to the no-GIL mode.
jillesvangurp 3y ago
That's fine. We go from suspecting there are issues to knowing exactly where the issues are. The rest is just chipping away at that list and making the issues go away. Either by adding some kind of mutex around the code or by replacing the native code with something less likely to have issues. The argument against this seems mainly that it's a lot of work; not that it's impossible work. It probably is a lot of work but if there are enough people doing the work, we should get some results.
mkoubaa 3y ago
Might be a good time to write concurrency first C extension to compete with the legacy ones
xmcqdpt2 3y ago
> Note that the code may cause a context switch even today if the C code callbacks into Python bytecode (via a __del__ method) and the bytecode is long enough (100 instructions I think). However, that is extremely unlikely and much C extension code is not written with such situations in mind. As someone who works professionally on a parallel / async runtime that supports thousands of continuously running servers, "extremely unlikely" means that actually it breaks all the time, but it's also impossible to debug.
deleted 3y ago
[deleted]

qbasic_forever 3y ago

Yep there are a ton of issues like that to be found, and unfortunately they will manifest as difficult to find and debug race conditions. This is why the proposal and work is to make non-GIL mode entirely optional and not the default. It just means for the brave few that flip it on and use it, be prepared to spend a huge amount of time finding and fixing subtle race conditions in decades of old python library code. The early adopters are going to be in for a lot of pain, or more likely they'll restrict their use of non-GIL processes to very specialized and dedicated processes that have as few dependencies as possible.

miraculixx 3y ago
The intent is to make no GIL the default eventually.
- n2d4 3y ago
  I don't think this is true. There are fairly strong voices on both sides inside the community, at this time it's pretty uncertain. To quote Guido: >Let’s not blow it this time. If we’re going forward with nogil (and I’m not saying we are, but I can’t exclude it), let’s make sure there is a way to be able to import extensions requiring the GIL in a nogil interpreter without any additional shenanigans https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional-3-12-updates/26503/19 https://discuss.python.org/t/pep-703-making-the-global-inter...
  - Alphaeus 3y ago
    The Steering Council said their intention is to remove the GIL-build in future: > Long-term, we want no-GIL to be the default, and to remove any vestiges of the GIL (without unnecessarily breaking backward compatibility). We don’t want to wait too long with this, because having two common build modes may be a heavy burden on the community (as, for example, it can double test resources and debugging scenarios), but we can’t rush it either. We think it may take as much as five years to get to this stage. https://discuss.python.org/t/a-steering-council-notice-about-pep-703-making-the-global-interpreter-lock-optional-in-cpython/30474 https://discuss.python.org/t/a-steering-council-notice-about...
    - toyg 3y ago
      By the usual laws of software estimates, if they think it will take 5 years, it's going to be more like 10-15.
hinkley 3y ago
Does Python have a lot of secondary dependencies? I could see someone pulling in two dependencies, not realize they both use the same unsafe library, and end up having them step all over each other.
- qbasic_forever 3y ago
  It does, so much so things like virtualenv were introduced so every program can have its own set of dependencies such that they won't clash with other libs on your system. Something like flask or fastapi pull in a lot of secondary dependencies alone.
  - hinkley 3y ago
    Probably doesn't work with FFI though? I'm trying to get a sense of how hypothetical versus very real this problem is.

smcl 3y ago

I think the CPython core devs are very keenly aware of these issues. Otherwise they'd have announced a plan to suddenly rip out the GIL altogether, rather than a phased approach that allows people to opt-in to the no-GIL mode.

jillesvangurp 3y ago

That's fine. We go from suspecting there are issues to knowing exactly where the issues are. The rest is just chipping away at that list and making the issues go away. Either by adding some kind of mutex around the code or by replacing the native code with something less likely to have issues. The argument against this seems mainly that it's a lot of work; not that it's impossible work. It probably is a lot of work but if there are enough people doing the work, we should get some results.

mkoubaa 3y ago

Might be a good time to write concurrency first C extension to compete with the legacy ones

xmcqdpt2 3y ago

> Note that the code may cause a context switch even today if the C code callbacks into Python bytecode (via a __del__ method) and the bytecode is long enough (100 instructions I think). However, that is extremely unlikely and much C extension code is not written with such situations in mind. As someone who works professionally on a parallel / async runtime that supports thousands of continuously running servers, "extremely unlikely" means that actually it breaks all the time, but it's also impossible to debug.

deleted 3y ago

[deleted]