3 ms·

I dont get this. Just because you don't have the GIL doesnt mean that your previously single threaded code is now multithreaded and stepping on itself.

411111111111111 3y ago
From previous discussions it's my understanding the c integration is going to be the cause for the issues. From a python perspective it wouldn't necessarily be a big change, but everything can branch out to c, and there you're going to get in trouble with shared memory.
- dwaite 3y ago
  The most significant impact of this change is that python threads can run at the same time. Before, calls to C API were essentially cooperative multithreading yield points - even if you have 100 python threads, they can't run concurrently under the GIL. 99 are blocked waiting on the GIL. C extensions have always been able to use real threading. But now there is no GIL to synchronize with the python interpreter on ingress/egress from the extension.
deleted 3y ago
[deleted]
T-A 3y ago
No, but it means that your previous multithreaded code is no longer automatically prevented from stepping all over itself by having multiple threads accessing the same data: https://realpython.com/python-gil/#what-problem-did-the-gil-solve-for-python https://realpython.com/python-gil/#what-problem-did-the-gil-...
- agf 3y ago
  GIL removal doesn't mean "make all of this lockless" it means "replace the GIL with fine-grained locking". So those problems are still solved for Python code. The three issues are the amount of work it takes to do right, the performance cost for single-threaded code, and the CAPI.
  - tomn 3y ago
    finally. you don't even have to read anything to work this out -- if things like dictionary access were no longer atomic that would imply that threaded code without locks could crash the interpreter, which isn't going to happen.
  - hughesjj 3y ago
    I'm simultaneously scared and enlightened seeing all these comments acting as if the GIL is some magic "makes your code thread/concurrency safe" pancea. I always saw it as a shim/hack to make cpython specifically easier to implement, not something that inherently makes code more thread safe or atomic. It's just more work to do things "the right way" across application boundaries, but from my understanding this PEP is Meta commiting to do that work.
    - kzrdude 3y ago
      Removing the lock creates problems in existing code in practice. This is an ecosystem that has less focus on standards and more on "CPython is the reference implementation".
      - hughesjj 3y ago
        What non-transparent GIL specific behavior are developers relying on exactly? When I say GIL specific behavior, I mean "python code that specifically requires a GLOBAL interpreter lock to function properly" Not something that simply requires atomic access or any of the garuntees that the GIL has this far provided, but like, specifically code that requires GIL like behavior above any CPython implementation details that could be implemented with more fine grained concurrency assurances? I've seen some really cursed python in my days, like checking against `locals()` to see if a variable was defined ala JavaScript's 'foo in window' syntax (but I suppose more portable), but I can't recall anything specifically caring about a global interpreter lock (instead of what the GIL has semantically provided, which is much different)
        dwaite 3y ago
- hanniabu 3y ago
  Why cant they just have a flag to enable it? Or something in the file head similar to shebang
  - schmeii 3y ago
    There is a flag to enable it, and the default is to run with the GIL.

411111111111111 3y ago

From previous discussions it's my understanding the c integration is going to be the cause for the issues. From a python perspective it wouldn't necessarily be a big change, but everything can branch out to c, and there you're going to get in trouble with shared memory.

dwaite 3y ago
The most significant impact of this change is that python threads can run at the same time. Before, calls to C API were essentially cooperative multithreading yield points - even if you have 100 python threads, they can't run concurrently under the GIL. 99 are blocked waiting on the GIL. C extensions have always been able to use real threading. But now there is no GIL to synchronize with the python interpreter on ingress/egress from the extension.

deleted 3y ago

[deleted]

T-A 3y ago

No, but it means that your previous multithreaded code is no longer automatically prevented from stepping all over itself by having multiple threads accessing the same data: https://realpython.com/python-gil/#what-problem-did-the-gil-solve-for-python https://realpython.com/python-gil/#what-problem-did-the-gil-...

agf 3y ago
GIL removal doesn't mean "make all of this lockless" it means "replace the GIL with fine-grained locking". So those problems are still solved for Python code. The three issues are the amount of work it takes to do right, the performance cost for single-threaded code, and the CAPI.
- tomn 3y ago
  finally. you don't even have to read anything to work this out -- if things like dictionary access were no longer atomic that would imply that threaded code without locks could crash the interpreter, which isn't going to happen.
- hughesjj 3y ago
  I'm simultaneously scared and enlightened seeing all these comments acting as if the GIL is some magic "makes your code thread/concurrency safe" pancea. I always saw it as a shim/hack to make cpython specifically easier to implement, not something that inherently makes code more thread safe or atomic. It's just more work to do things "the right way" across application boundaries, but from my understanding this PEP is Meta commiting to do that work.
  - kzrdude 3y ago
    Removing the lock creates problems in existing code in practice. This is an ecosystem that has less focus on standards and more on "CPython is the reference implementation".
    - hughesjj 3y ago
      What non-transparent GIL specific behavior are developers relying on exactly? When I say GIL specific behavior, I mean "python code that specifically requires a GLOBAL interpreter lock to function properly" Not something that simply requires atomic access or any of the garuntees that the GIL has this far provided, but like, specifically code that requires GIL like behavior above any CPython implementation details that could be implemented with more fine grained concurrency assurances? I've seen some really cursed python in my days, like checking against `locals()` to see if a variable was defined ala JavaScript's 'foo in window' syntax (but I suppose more portable), but I can't recall anything specifically caring about a global interpreter lock (instead of what the GIL has semantically provided, which is much different)
      - dwaite 3y ago
hanniabu 3y ago
Why cant they just have a flag to enable it? Or something in the file head similar to shebang
- schmeii 3y ago
  There is a flag to enable it, and the default is to run with the GIL.