7 ms·

"It cannot be understated, removing the GIL will be a HUGE deal for Python! It's an ugly wart which has existed for about 30 years, and nobody else has produced

"It cannot be understated, removing the GIL will be a HUGE deal for Python! It's an ugly wart which has existed for about 30 years, and nobody else has produced and delivered a working solution to the community." If it's such a huge game changer why don't some of the large enterprises which rely on Python fund this work?

deleted 4y ago
[deleted]
meta_throwaway 4y ago
that's exactly what has happened. edit: not sure why I got flagged, HN. Software engineer hours == funding.
ianai 4y ago
Well, I might prefer a gofundme for some devs to do it. That way us average Joe’s can pitch into the support. Otherwise, there’s making a fork and attempting periodic updates from python. That’d be a huge undertaking though. FTA: “”” A lot of the value of Python is the ecosystem, not just the language… CPython really leads the way in terms of the community moving as a block. "Removing the GIL is a really transformative step. Most Python programs just don’t use threads at the moment if they want to run on multiple cores. If nogil is to be a success, the community as a whole has to buy into it." – Sam Gross”””
Scarbutt 4y ago
Because Java,C#,Golang,Javascript exist.
- mateo411 4y ago
  Wait, does Javascript have good multi-threading support? I dont think it does.
  - mftb 4y ago
    Worker threads[0]. If you don't follow Javascript it's not surprising you hadn't heard about it, it's pretty new. Node also, thanks to v8, pretty much outperforms Python. [0]https://nodejs.org/api/worker_threads.html https://nodejs.org/api/worker_threads.html
    - frazbin 4y ago
      worker threads don't share the parent's heap though, right? They can only share state through a binary channel. It's something, but pretty different from traditional pthreads.
      - mftb 4y ago
        They can use SharedArrayBuffers[0] which wrap shared memory or they can pass ArrayBuffers. Node may not have the shared arrays yet, but the browsers do. [0]https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
        ptx 4y ago
        Python has multiprocessing.shared_memory[1] since 3.8 to share arrays between processes, which accomplishes the same thing and circumvents the GIL issue. [1] https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem...
gorgoiler 4y ago
Remember when it was discovered that arc welders could produce Carbon 60, and there was a huge run on arc welders? All of a sudden there was a whole new class of customer beyond the usual customer base of people who do welding: people who do materials science research. I would hazard a guess to say that 99% of production Python is people doing the equivalent of welding, but there is also this 1% who want the GIL to be gone so they can repurpose Python as a tool for a whole new class of problem solving. It will be an exciting future for them. For us welders, we’ll carry on gluing stuff together blissfully unaware of the GIL. Async and non-blocking IO are great. I don’t think I’ve ever needed compute-concurrency, not in Python anyway.
- colinmhayes 4y ago
  Eh, even with true multithreading python will still be slow as molasses. If you want speed python will still not be your choice.
  - Proven 4y ago
  - saghm 4y ago
    I'm not sure I fully buy this argument. I think programming languages are chosen for projects based on a number of different factors, speed being one of them, but developer productivity being another (and there are certainly many more). Given that there isn't usually going to be one language that is the best choice for every single factor, it seems pretty natural that some teams might pick Python due to productivity being more important but still have some need for better performance, and other teams might not be able to compromise on performance and have to resort to something like Java or Go or C++ but still would be able to iterate faster with something higher level. It's definitely not a given that there are enough potential projects that would get enough benefit from removing the GIL to make it worth it, but it seems silly to claim that anyone would would get any possible benefit from performance would never choose Python for other reasons.
    - 4y ago
latenightcoding 4y ago
Because it's not a huge game changer. What would you do differently if the GIL was gone? when people need speed in Python they write C/C++ extensions or use multiple processes/computers
- thelastbender12 4y ago
  >Because it's not a huge game changer. That is just not true. 1 - This helps parallelize regular python code too, like the kind that goes into writing web services. 2 - While you'd still write native extensions to get optimum performance for single threaded code, having the thread dispatch layer in python makes parallelizing execution convenient and robust. For example, see the comment from people who maintain scikit-learn below. I'd love to see python get composable task parallelism like Julia where you can freely spawn threads within threads without worrying about their lifetimes. link: https://github.com/colesbury/nogil/issues/36#issuecomment-1034624897 https://github.com/colesbury/nogil/issues/36#issuecomment-10...
  - diffeomorphism 4y ago
    That does not sound all that enthusiastic for a "game changer": > helps parallelize > convenient and robust So nice but not too noticeable?
    - diordiderot 4y ago
      Well context helps determine impact. What percentage of compute time is python? What percent of Py compute time would be cut? Who's using python?(banks, stem researchers, everyone else)
      - diffeomorphism 4y ago
        That has nothing to do with what I wrote and does not address the question? > ...bunch of questions... Yeah, if you claim that something is a "game changer" you should provide some arguments, not questions/"help determine". It might very well be a game changer but thud far you have only suggested there might be an argument for that, if ...
dralley 4y ago
I don't know if this is still true, but a few years ago there was only a single Python core developer that worked on it full time (Victor Stinner from Red Hat). The rest were only able to devote somewhere between few hours to one day a week. https://discuss.python.org/t/official-list-of-core-developers/924 https://discuss.python.org/t/official-list-of-core-developer...
- di 4y ago
  Łukasz Langa now works full time on CPython via the PSF's Developer in Residence program. https://pyfound.blogspot.com/2021/07/ukasz-langa-is-inaugural-cpython.html https://pyfound.blogspot.com/2021/07/ukasz-langa-is-inaugura...
  - dralley 4y ago
    Good to hear!
- waygood 4y ago
  Microsoft has also been funding a team to work full-time on making Python faster: https://developers.slashdot.org/story/21/05/17/0225252/microsoft-funds-a-team-with-guido-van-rossum-to-double-the-speed-of-python https://developers.slashdot.org/story/21/05/17/0225252/micro...
jreese 4y ago
Sam Gross works for Meta, and to my knowledge, this is his #1 project. So in effect, one of the largest companies depending on Python is funding this work.
dataangel 4y ago
They've tried several times, see Unladen Swallow, but usually the projects are way more ambitious and also trying to introduce JIT.
saghm 4y ago
It doesn't sound like the reservations of the people in charge of Python are about funding; I don't think it's as simple as "pay them $1m to merge the PR and then pay them salaries as they maintain it". Unless the assumption is that the people in charge of Python would take a bribe, I don't see how large companies could somehow magically make this happen with money short of just forking Python and trying to maintain it themselves, which would burn a ton of goodwill and likely not even win over the community.
imron 4y ago
> If it's such a huge game changer why don't some of the large enterprises which rely on Python fund this work? Because lots of code in python relies on the GIL to act as a synchronization primitive for threaded code (either explicitly or implicitly) and removing the GIL will break that code in subtle ways and there’s no appetite in those companies to fix that.
22SAS 4y ago
My thoughts exactly. Google has already done quite a lot of work on LLVM, as an example. Python is among their most used programming languages, this something they could totally assemble engineers to do.

deleted 4y ago

[deleted]

meta_throwaway 4y ago

that's exactly what has happened. edit: not sure why I got flagged, HN. Software engineer hours == funding.

ianai 4y ago

Well, I might prefer a gofundme for some devs to do it. That way us average Joe’s can pitch into the support. Otherwise, there’s making a fork and attempting periodic updates from python. That’d be a huge undertaking though. FTA: “”” A lot of the value of Python is the ecosystem, not just the language… CPython really leads the way in terms of the community moving as a block. "Removing the GIL is a really transformative step. Most Python programs just don’t use threads at the moment if they want to run on multiple cores. If nogil is to be a success, the community as a whole has to buy into it." – Sam Gross”””

Scarbutt 4y ago

Because Java,C#,Golang,Javascript exist.

mateo411 4y ago
Wait, does Javascript have good multi-threading support? I dont think it does.
- mftb 4y ago
  Worker threads[0]. If you don't follow Javascript it's not surprising you hadn't heard about it, it's pretty new. Node also, thanks to v8, pretty much outperforms Python. [0]https://nodejs.org/api/worker_threads.html https://nodejs.org/api/worker_threads.html
  - frazbin 4y ago
    worker threads don't share the parent's heap though, right? They can only share state through a binary channel. It's something, but pretty different from traditional pthreads.
    - mftb 4y ago
      They can use SharedArrayBuffers[0] which wrap shared memory or they can pass ArrayBuffers. Node may not have the shared arrays yet, but the browsers do. [0]https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
      - ptx 4y ago
        Python has multiprocessing.shared_memory[1] since 3.8 to share arrays between processes, which accomplishes the same thing and circumvents the GIL issue. [1] https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem...

gorgoiler 4y ago

Remember when it was discovered that arc welders could produce Carbon 60, and there was a huge run on arc welders? All of a sudden there was a whole new class of customer beyond the usual customer base of people who do welding: people who do materials science research. I would hazard a guess to say that 99% of production Python is people doing the equivalent of welding, but there is also this 1% who want the GIL to be gone so they can repurpose Python as a tool for a whole new class of problem solving. It will be an exciting future for them. For us welders, we’ll carry on gluing stuff together blissfully unaware of the GIL. Async and non-blocking IO are great. I don’t think I’ve ever needed compute-concurrency, not in Python anyway.

colinmhayes 4y ago
Eh, even with true multithreading python will still be slow as molasses. If you want speed python will still not be your choice.
- Proven 4y ago
- saghm 4y ago
  I'm not sure I fully buy this argument. I think programming languages are chosen for projects based on a number of different factors, speed being one of them, but developer productivity being another (and there are certainly many more). Given that there isn't usually going to be one language that is the best choice for every single factor, it seems pretty natural that some teams might pick Python due to productivity being more important but still have some need for better performance, and other teams might not be able to compromise on performance and have to resort to something like Java or Go or C++ but still would be able to iterate faster with something higher level. It's definitely not a given that there are enough potential projects that would get enough benefit from removing the GIL to make it worth it, but it seems silly to claim that anyone would would get any possible benefit from performance would never choose Python for other reasons.
  - 4y ago

latenightcoding 4y ago

Because it's not a huge game changer. What would you do differently if the GIL was gone? when people need speed in Python they write C/C++ extensions or use multiple processes/computers

thelastbender12 4y ago
>Because it's not a huge game changer. That is just not true. 1 - This helps parallelize regular python code too, like the kind that goes into writing web services. 2 - While you'd still write native extensions to get optimum performance for single threaded code, having the thread dispatch layer in python makes parallelizing execution convenient and robust. For example, see the comment from people who maintain scikit-learn below. I'd love to see python get composable task parallelism like Julia where you can freely spawn threads within threads without worrying about their lifetimes. link: https://github.com/colesbury/nogil/issues/36#issuecomment-1034624897 https://github.com/colesbury/nogil/issues/36#issuecomment-10...
- diffeomorphism 4y ago
  That does not sound all that enthusiastic for a "game changer": > helps parallelize > convenient and robust So nice but not too noticeable?
  - diordiderot 4y ago
    Well context helps determine impact. What percentage of compute time is python? What percent of Py compute time would be cut? Who's using python?(banks, stem researchers, everyone else)
    - diffeomorphism 4y ago
      That has nothing to do with what I wrote and does not address the question? > ...bunch of questions... Yeah, if you claim that something is a "game changer" you should provide some arguments, not questions/"help determine". It might very well be a game changer but thud far you have only suggested there might be an argument for that, if ...

dralley 4y ago

I don't know if this is still true, but a few years ago there was only a single Python core developer that worked on it full time (Victor Stinner from Red Hat). The rest were only able to devote somewhere between few hours to one day a week. https://discuss.python.org/t/official-list-of-core-developers/924 https://discuss.python.org/t/official-list-of-core-developer...

di 4y ago
Łukasz Langa now works full time on CPython via the PSF's Developer in Residence program. https://pyfound.blogspot.com/2021/07/ukasz-langa-is-inaugural-cpython.html https://pyfound.blogspot.com/2021/07/ukasz-langa-is-inaugura...
- dralley 4y ago
  Good to hear!
waygood 4y ago
Microsoft has also been funding a team to work full-time on making Python faster: https://developers.slashdot.org/story/21/05/17/0225252/microsoft-funds-a-team-with-guido-van-rossum-to-double-the-speed-of-python https://developers.slashdot.org/story/21/05/17/0225252/micro...

jreese 4y ago

Sam Gross works for Meta, and to my knowledge, this is his #1 project. So in effect, one of the largest companies depending on Python is funding this work.

dataangel 4y ago

They've tried several times, see Unladen Swallow, but usually the projects are way more ambitious and also trying to introduce JIT.

saghm 4y ago

It doesn't sound like the reservations of the people in charge of Python are about funding; I don't think it's as simple as "pay them $1m to merge the PR and then pay them salaries as they maintain it". Unless the assumption is that the people in charge of Python would take a bribe, I don't see how large companies could somehow magically make this happen with money short of just forking Python and trying to maintain it themselves, which would burn a ton of goodwill and likely not even win over the community.

imron 4y ago

> If it's such a huge game changer why don't some of the large enterprises which rely on Python fund this work? Because lots of code in python relies on the GIL to act as a synchronization primitive for threaded code (either explicitly or implicitly) and removing the GIL will break that code in subtle ways and there’s no appetite in those companies to fix that.

22SAS 4y ago

My thoughts exactly. Google has already done quite a lot of work on LLVM, as an example. Python is among their most used programming languages, this something they could totally assemble engineers to do.