4 ms·

As I understand it, if your code would have race conditions with free threaded python, than it probably already has them.

stavros 2y ago
Not when there's a global interpreter lock.
- pdonis 2y ago
  The GIL does not prevent race conditions in your Python code. It only prevents race conditions in internal data structures inside the interpreter and in atomic operations, i.e., operations that take a single Python bytecode. But many things that appear atomic in Python code take more than one Python bytecode. The GIL gives you no protection if you do such operations in multiple threads on the same object.
  - stavros 2y ago
    I answered in a sibling reply: https://news.ycombinator.com/item?id=40950798 https://news.ycombinator.com/item?id=40950798
    - pdonis 2y ago
      I'll respond there.
      - vulnbludog 2y ago
        I’m a dummy don’t know nothing about coding but I love HN usernames lol
  - matsemann 2y ago
    I think what many will experience, is that they want to switch to multithreading without GIL, but learn they have code that will have race conditions. But that don't have race conditions today, as it's run as multiple processes, and not threads. For instance our webserver. It uses multiple processes. Each request then can modify some global variable, use as cache or whatever, and only after it's completely done handling the request the same process will serve a new request. But when people see the GIL is gone, they probably would like to start using it. Can handle more requests without spamming processes / using beefy computer with lots of RAM etc. And then one might discover new race conditions one never really had before.
- dagenix 2y ago
  Are you writing an extension or Python code? If you are writing Python code, the GIL can already be dropped at pretty much any point and there isn't much way of controlling when. Iirc, this includes in the middle of things like +=. There are some operations that Python defines as atomic, but, as I recall, there aren't all that many. In what way is the GIL preventing races for your use case?
  - stavros 2y ago
    I mean that, if the GIL didn't prevent races, it would be trivially removable. Races that are already there in people's Python code have probably been debugged (or at least they are tolerated), so there are some races that will happen when the GIL is removed, and they will be a surprise.
    - dagenix 2y ago
      The GIL prevents the corruption of Pythons internal structures. It's hard to remove because: 1. Lots of extensions, which can control when they release the GIL unlike regular Python code, depend on it 2. Removing the GIL requires some sort of other mechanism to protect internal Python stuff 3. But for a long time, such a mechanism was resisted by th Python team because all attempts to remove the GIL either made single threaded code slower or were considered too complicated. But, as far as I understand, the GIL does somewhere between nothing and very little to prevent races in pure Python code. And, my rough understanding, is that removing the GIL isn't expected to really impact pure Python code.
      - stavros 2y ago
        Hmm, that's interesting, thank you. I didn't realize extensions can control the GIL.
        dagenix 2y ago
        Yup, I think its described here: https://docs.python.org/3/c-api/init.html#releasing-the-gil-from-extension-code https://docs.python.org/3/c-api/init.html#releasing-the-gil-.... My understanding, is that many extensions will release the GIL when doing anything expensive. So, if you are doing CPU or IO bound operations in an extension _and_ you are calling that operation in multiple threads, even with the GIL you can potentially fully utilize all of the CPUs in your machine.

stavros 2y ago

Not when there's a global interpreter lock.

pdonis 2y ago
The GIL does not prevent race conditions in your Python code. It only prevents race conditions in internal data structures inside the interpreter and in atomic operations, i.e., operations that take a single Python bytecode. But many things that appear atomic in Python code take more than one Python bytecode. The GIL gives you no protection if you do such operations in multiple threads on the same object.
- stavros 2y ago
  I answered in a sibling reply: https://news.ycombinator.com/item?id=40950798 https://news.ycombinator.com/item?id=40950798
  - pdonis 2y ago
    I'll respond there.
    - vulnbludog 2y ago
      I’m a dummy don’t know nothing about coding but I love HN usernames lol
- matsemann 2y ago
  I think what many will experience, is that they want to switch to multithreading without GIL, but learn they have code that will have race conditions. But that don't have race conditions today, as it's run as multiple processes, and not threads. For instance our webserver. It uses multiple processes. Each request then can modify some global variable, use as cache or whatever, and only after it's completely done handling the request the same process will serve a new request. But when people see the GIL is gone, they probably would like to start using it. Can handle more requests without spamming processes / using beefy computer with lots of RAM etc. And then one might discover new race conditions one never really had before.
dagenix 2y ago
Are you writing an extension or Python code? If you are writing Python code, the GIL can already be dropped at pretty much any point and there isn't much way of controlling when. Iirc, this includes in the middle of things like +=. There are some operations that Python defines as atomic, but, as I recall, there aren't all that many. In what way is the GIL preventing races for your use case?
- stavros 2y ago
  I mean that, if the GIL didn't prevent races, it would be trivially removable. Races that are already there in people's Python code have probably been debugged (or at least they are tolerated), so there are some races that will happen when the GIL is removed, and they will be a surprise.
  - dagenix 2y ago
    The GIL prevents the corruption of Pythons internal structures. It's hard to remove because: 1. Lots of extensions, which can control when they release the GIL unlike regular Python code, depend on it 2. Removing the GIL requires some sort of other mechanism to protect internal Python stuff 3. But for a long time, such a mechanism was resisted by th Python team because all attempts to remove the GIL either made single threaded code slower or were considered too complicated. But, as far as I understand, the GIL does somewhere between nothing and very little to prevent races in pure Python code. And, my rough understanding, is that removing the GIL isn't expected to really impact pure Python code.
    - stavros 2y ago
      Hmm, that's interesting, thank you. I didn't realize extensions can control the GIL.
      - dagenix 2y ago
        Yup, I think its described here: https://docs.python.org/3/c-api/init.html#releasing-the-gil-from-extension-code https://docs.python.org/3/c-api/init.html#releasing-the-gil-.... My understanding, is that many extensions will release the GIL when doing anything expensive. So, if you are doing CPU or IO bound operations in an extension _and_ you are calling that operation in multiple threads, even with the GIL you can potentially fully utilize all of the CPUs in your machine.