4 ms·

The biggest hurdle I think is still the GIL. Does anyone know if any progress is being made there? https://wiki.python.org/moin/GlobalInterpreterLock https://w

The biggest hurdle I think is still the GIL. Does anyone know if any progress is being made there? https://wiki.python.org/moin/GlobalInterpreterLock https://wiki.python.org/moin/GlobalInterpreterLock

oblio 4y ago
I think there were some initiatives, but I don't know if they're still ongoing. Unladen Swallow from Google didn't succeed.
morelisp 4y ago
I believe the plan is still that the improvements in 3.11 are to be traded off against the overhead of eventual GIL removal. So, don't enjoy them too much.
mkoubaa 4y ago
Gil free python would break a lot of oss packages written with the C API that are only thread safe because if the GIL. Couple that with the fact that python on its own without packages isn't that useful and you can see why this is going to be a challenge . Multiple interpreters in the same process with their own locks is happening now, though
- lzauz 4y ago
  >Gil free python would break a lot of oss packages written with the C API that are only thread safe because if the GIL. So make it a runtime option. It's getting tiresome that python performance is suffering because some can't be bothered to write thread-safe software. In 2022.
  - coldtea 4y ago
    In C which Python extensions use, it hasen't gotten any easier to write thread-safe software "in 2022"
    - ahartmetz 4y ago
      But it has become more necessary and useful, so even C programmers better get used to it (and some tooling to help catch most mistakes).
      - coldtea 4y ago
        >so even C programmers better get used to it Or else? It's not like they're not trying their best - or don't spend the level of effort that they and their companies are willing to take...
        ahartmetz 4y ago
        I guess it would have made more sense for me to say "Python C extension developers". I don't find multithreading in languages without particular support easy at all, but I have become better at it. It is possible and sometimes necessary. It seems like the prevailing attitude in the Python ecosystem is weird, a kind of sour grapes thing, i.e. "Python doesn't have good multithreading support, but multithreading is ugly and error-prone anyway and the alternatives are almost as good or better".
  - AdamN 4y ago
    Usually when I've needed more parallelization I've allowed more processes and for slow methods, there is threading available (this doesn't overcome the GIL but allows those methods to independently operate). It seems like the biggest reasons to focus on removing the GIL are single-process applications or machines where memory is constrained (so you don't want tons of processes consuming it all). Are you in one of those situations or is there another scenario that is impacted by the GIL?
  - kortex 4y ago
    This comment demonstrates a tremendous amount of naivete about the Cpython runtime. After PyObject itself, the GIL mutex is probably the next most important data structure in the entire codebase. It's not "someone not being bothered to write thread-safe software." It's not something you can hide behind a flag. It's central to the entire cpython data model and any library which relies on releasing the GIL. The closest anyone has come to removing the GIL is the Gilectomy project by Larry Hastings, and it's unlikely to ever be upstreamed unless it could be somehow made to work with libraries that rely on assumptions about GIL mechanics (eg numpy).
    - mixmastamyk 4y ago
      > The closest anyone has come to removing the GIL is the Gilectomy project by Larry Hastings It was Sam Gross and he more or less achieved it: https://mail.python.org/archives/list/python-dev@python.org/thread/ABR2L6BENNA6UPSPKV474HCS4LWT26GY/ https://mail.python.org/archives/list/python-dev@python.org/...
- ptx 4y ago
  Wouldn't it also break most Python code that uses threading? Currently, everything is implicitly synchronized thanks to the GIL, so in Java terms it's as if every variable is declared "volatile". If the GIL is removed and variables are no longer volatile (i.e, changes are no longer made visible immediately to other threads) that seems like it would break a lot of code. On the other hand, keeping every variable volatile seems like it would be terrible for performance. Maybe I'm missing some critical difference between Python and the JVM here?
  - kortex 4y ago
    The GILectomy project was able to maintain thread safety on refcounting while (mostly) preserving performance by basically using some clever flags in the refcount to signal specific lifetimes/ownership (simplification for brevity). So you have good performance when the thread owns a reference while preserving safety for shared objects. The smaller problem is this does at a small amount of overhead even to single-threaded performance. The gilectomy project improved performance elsewhere so net performance is close to the same. The bigger problem would be integrating this strategy with all the libraries that rely on existing GIL behavior.
sweezyjeezy 4y ago
https://m.youtube.com/watch?t=7960 https://m.youtube.com/watch?t=7960 Guido discussing this recently on Lex Friedman's podcast. tl;dw, he is open to the idea, he thinks it will be painful, he isn't convinced the demand is high enough yet .
martin_loetzsch 4y ago
I think this summarizes the topic quite well: https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-python_11.html https://pyfound.blogspot.com/2022/05/the-2022-python-languag...
theandrewbailey 4y ago
Sidestep it by using multiple interpreters, each with their own GIL: https://peps.python.org/pep-0554/ https://peps.python.org/pep-0554/ https://peps.python.org/pep-0684/ https://peps.python.org/pep-0684/

oblio 4y ago

I think there were some initiatives, but I don't know if they're still ongoing. Unladen Swallow from Google didn't succeed.

morelisp 4y ago

I believe the plan is still that the improvements in 3.11 are to be traded off against the overhead of eventual GIL removal. So, don't enjoy them too much.

mkoubaa 4y ago

Gil free python would break a lot of oss packages written with the C API that are only thread safe because if the GIL. Couple that with the fact that python on its own without packages isn't that useful and you can see why this is going to be a challenge . Multiple interpreters in the same process with their own locks is happening now, though

lzauz 4y ago
>Gil free python would break a lot of oss packages written with the C API that are only thread safe because if the GIL. So make it a runtime option. It's getting tiresome that python performance is suffering because some can't be bothered to write thread-safe software. In 2022.
- coldtea 4y ago
  In C which Python extensions use, it hasen't gotten any easier to write thread-safe software "in 2022"
  - ahartmetz 4y ago
    But it has become more necessary and useful, so even C programmers better get used to it (and some tooling to help catch most mistakes).
    - coldtea 4y ago
      >so even C programmers better get used to it Or else? It's not like they're not trying their best - or don't spend the level of effort that they and their companies are willing to take...
      - ahartmetz 4y ago
        I guess it would have made more sense for me to say "Python C extension developers". I don't find multithreading in languages without particular support easy at all, but I have become better at it. It is possible and sometimes necessary. It seems like the prevailing attitude in the Python ecosystem is weird, a kind of sour grapes thing, i.e. "Python doesn't have good multithreading support, but multithreading is ugly and error-prone anyway and the alternatives are almost as good or better".
- AdamN 4y ago
  Usually when I've needed more parallelization I've allowed more processes and for slow methods, there is threading available (this doesn't overcome the GIL but allows those methods to independently operate). It seems like the biggest reasons to focus on removing the GIL are single-process applications or machines where memory is constrained (so you don't want tons of processes consuming it all). Are you in one of those situations or is there another scenario that is impacted by the GIL?
- kortex 4y ago
  This comment demonstrates a tremendous amount of naivete about the Cpython runtime. After PyObject itself, the GIL mutex is probably the next most important data structure in the entire codebase. It's not "someone not being bothered to write thread-safe software." It's not something you can hide behind a flag. It's central to the entire cpython data model and any library which relies on releasing the GIL. The closest anyone has come to removing the GIL is the Gilectomy project by Larry Hastings, and it's unlikely to ever be upstreamed unless it could be somehow made to work with libraries that rely on assumptions about GIL mechanics (eg numpy).
  - mixmastamyk 4y ago
    > The closest anyone has come to removing the GIL is the Gilectomy project by Larry Hastings It was Sam Gross and he more or less achieved it: https://mail.python.org/archives/list/python-dev@python.org/thread/ABR2L6BENNA6UPSPKV474HCS4LWT26GY/ https://mail.python.org/archives/list/python-dev@python.org/...
ptx 4y ago
Wouldn't it also break most Python code that uses threading? Currently, everything is implicitly synchronized thanks to the GIL, so in Java terms it's as if every variable is declared "volatile". If the GIL is removed and variables are no longer volatile (i.e, changes are no longer made visible immediately to other threads) that seems like it would break a lot of code. On the other hand, keeping every variable volatile seems like it would be terrible for performance. Maybe I'm missing some critical difference between Python and the JVM here?
- kortex 4y ago
  The GILectomy project was able to maintain thread safety on refcounting while (mostly) preserving performance by basically using some clever flags in the refcount to signal specific lifetimes/ownership (simplification for brevity). So you have good performance when the thread owns a reference while preserving safety for shared objects. The smaller problem is this does at a small amount of overhead even to single-threaded performance. The gilectomy project improved performance elsewhere so net performance is close to the same. The bigger problem would be integrating this strategy with all the libraries that rely on existing GIL behavior.

sweezyjeezy 4y ago

https://m.youtube.com/watch?t=7960 https://m.youtube.com/watch?t=7960 Guido discussing this recently on Lex Friedman's podcast. tl;dw, he is open to the idea, he thinks it will be painful, he isn't convinced the demand is high enough yet .

martin_loetzsch 4y ago

I think this summarizes the topic quite well: https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-python_11.html https://pyfound.blogspot.com/2022/05/the-2022-python-languag...

theandrewbailey 4y ago

Sidestep it by using multiple interpreters, each with their own GIL: https://peps.python.org/pep-0554/ https://peps.python.org/pep-0554/ https://peps.python.org/pep-0684/ https://peps.python.org/pep-0684/