3 ms·

>> However, that leaves a lot of C code that you can't talk to anymore because the C code requires the old Python FFI. I think this is where the main problem li

>> However, that leaves a lot of C code that you can't talk to anymore because the C code requires the old Python FFI. I think this is where the main problem lies. This is exactly the problem, but people have a hard time grasping this because most people interacting with Python have no understanding of how C code interacts with Python, or don't understand the C module ecosystem. I'm not sure if the Python community has a good accounting of this either because I don't recall seeing much quantitative analysis of how many modules would need to be updated etc. This would help compare with the Python 2 to 3 conversion efforts. Even then, the site listing (shaming?) popular modules with compatibility made a mid-to-late appearance in the process of killing Python 2. Quantification of module updates is obvious thing to have from the get-go for anyone looking to follow through on removing the GIL, but it's not a fun task.

amelius 3y ago
This needs more thinking but how about a hybrid approach, where you have Thread objects, and GILFreeThread objects? The Thread objects work with old code, but run more slowly. The GILFreeThread objects are fast. If an object is passed from a Thread to a GILFreeThread or the other way around, then special safety code is attached to the object so that manipulating the object from the other side doesn't cause issues. The advantage is that now the module implementers have time to migrate from the old system to the new system. And users can work with both the old modules and "converted" modules in the same system, with minor changes.
- ptx 3y ago
  This sounds a bit like COM and its apartment-threaded vs. free-threaded objects. The "special safety code" in that case is a proxy object that sends messages to the thread that owns the actual object when its methods are invoked.
- kortex 3y ago
  That sounds like a maintenance and stability nightmare, if it's even possible. You are effectively red/blue splitting the entire codebase. PyObject and the GIL touch everything in the codebase.
  - amelius 3y ago
    The red/blue splitting happens behind the scenes, so it's different. Not really a color problem, because the user doesn't have to know about it. But yeah, you will basically have two versions of Python running at the same time, with some (hopefully invisible) translation between them.
    - kortex 3y ago
      > But the red/blue splitting happens behind the scenes, so it's different. Respectfully, I don't believe you have spent any appreciable time looking at the CPython source code. If you had, you would understand how unreasonable this expectation is. I don't say this to tear you down, I say this to convey the magnitude of what you are describing. It would involve touching tens of thousands of LoC. You are talking about a multi-million dollar project that would result in a ton of near-duplication of code. The red/blue is inescapable because you have to redefine PyObject to have two flavors, PyObject with GIL and GilFreePyObject. You now have to check which one you are dealing with constantly.
      - amelius 3y ago
        > You now have to check which one you are dealing with constantly. No, because if you're running inside a Thread you will know that you will see only PyObjects, whereas if you're running inside a GilFreeThread you will know that you will only see GilFreePyObjects. If you're manipulating the PyObject (necessarily from a Thread) then there will be behind-the-scenes translation code that will manipulate the corresponding GilFreePyObject for you. But you don't have to know about it.
        kortex 3y ago
        What exactly does "running inside a Thread/GilFreeThread" in the context of the cpython runtime mean? You pretty much need an entire copy of the virtual machine code. These are C structs we are talking about here, not some Rust trait you can readily parameterize over abstractly. That either means lots of manual code duplication, or some gnarly preprocessor action. Both are a maintenance nightmare.

amelius 3y ago

This needs more thinking but how about a hybrid approach, where you have Thread objects, and GILFreeThread objects? The Thread objects work with old code, but run more slowly. The GILFreeThread objects are fast. If an object is passed from a Thread to a GILFreeThread or the other way around, then special safety code is attached to the object so that manipulating the object from the other side doesn't cause issues. The advantage is that now the module implementers have time to migrate from the old system to the new system. And users can work with both the old modules and "converted" modules in the same system, with minor changes.

ptx 3y ago
This sounds a bit like COM and its apartment-threaded vs. free-threaded objects. The "special safety code" in that case is a proxy object that sends messages to the thread that owns the actual object when its methods are invoked.
kortex 3y ago
That sounds like a maintenance and stability nightmare, if it's even possible. You are effectively red/blue splitting the entire codebase. PyObject and the GIL touch everything in the codebase.
- amelius 3y ago
  The red/blue splitting happens behind the scenes, so it's different. Not really a color problem, because the user doesn't have to know about it. But yeah, you will basically have two versions of Python running at the same time, with some (hopefully invisible) translation between them.
  - kortex 3y ago
    > But the red/blue splitting happens behind the scenes, so it's different. Respectfully, I don't believe you have spent any appreciable time looking at the CPython source code. If you had, you would understand how unreasonable this expectation is. I don't say this to tear you down, I say this to convey the magnitude of what you are describing. It would involve touching tens of thousands of LoC. You are talking about a multi-million dollar project that would result in a ton of near-duplication of code. The red/blue is inescapable because you have to redefine PyObject to have two flavors, PyObject with GIL and GilFreePyObject. You now have to check which one you are dealing with constantly.
    - amelius 3y ago
      > You now have to check which one you are dealing with constantly. No, because if you're running inside a Thread you will know that you will see only PyObjects, whereas if you're running inside a GilFreeThread you will know that you will only see GilFreePyObjects. If you're manipulating the PyObject (necessarily from a Thread) then there will be behind-the-scenes translation code that will manipulate the corresponding GilFreePyObject for you. But you don't have to know about it.
      - kortex 3y ago
        What exactly does "running inside a Thread/GilFreeThread" in the context of the cpython runtime mean? You pretty much need an entire copy of the virtual machine code. These are C structs we are talking about here, not some Rust trait you can readily parameterize over abstractly. That either means lots of manual code duplication, or some gnarly preprocessor action. Both are a maintenance nightmare.