6 ms·

> That just is doing an incredible amount of heavy lifting. It'd be like saying, "Why doesn't the USA just switch entirely to the metric system?" It's a huge as

> That just is doing an incredible amount of heavy lifting. It'd be like saying, "Why doesn't the USA just switch entirely to the metric system?" It's a huge ask, and after being burned by the 2/3 transition, the python team is loathe to rock to boat too much again. > The GIL is deep, arguably one of the deepest abstractions in the codebase, up there with PyObject itself. Think about having to redefine the String implementation of a codebase in your language of choice. I am somewhat familiar with the cpython code base, and talk to some folks involves in some of thousand new python runtime. The problem is not that deep, and cpython is not that much different from other projects : the GIL is an implementation details that leaked through the API and now we have a bunch of people are relying on it. The question is what do we do about it. The trade-off in such situation are well known and just a question to pick the right one. > The team has already decided those tradeoffs are not really justifiable. That's from my view the crust of the issue, the TLDR is that GIL-less python is not a priority for the python team, so they view any trade-off/comprise as "not justifiable". Especially something when it come to added complexity to the code base or what not... > Person with limited to zero experience with CPython internals> I hate the GIL, why don't they just remove it? People have this reaction because having a single-threaded interpreter in 2023 is just ... embarrassing, what ever the reasons behind it.

nunuvit 3y ago
It only takes a couple of minutes to educate yourself on this topic. After reading it, let me know if you still think they don't want to compromise. https://www.artima.com/weblogs/viewpost.jsp?thread=214235 https://www.artima.com/weblogs/viewpost.jsp?thread=214235
- soulbadguy 3y ago
  > It only takes a couple of minutes to educate yourself on this topic. You can disagree without assuming that i am not familiar with the subject. The link you provided is from 2007 so i am not sure how fair it is to quote guido from then. But i am not sure the link demonstrate the point you are trying to make, from guido himself : > While it is my personal opinion, based upon the above considerations, that there isn't enough value in removing the GIL to warrant the effort. Edit : If you want a newer reference, this HN threads and the links discribe the situation well https://news.ycombinator.com/item?id=36240998 https://news.ycombinator.com/item?id=36240998 Which is the point i am making... This is not really a technical issue, the python just doesn't think it's worth the effort.
  - nunuvit 3y ago
    > From a steering council perspective we effectively view a 703 threading enabled interpreter at a high level as a fork of the CPython VM. "Fork" and "effort" are different problems. It's reductive to say that effort alone is the tldr. If the 2007 article and the linked current FAQ seem too old to be relevant, you're not really understanding what the steering committee is saying today.
    - soulbadguy 3y ago
      > "Fork" and "effort" are different problems. It's reductive to say that effort alone is the tldr. It's very unclear what you mean here, what does that have to do with the problem. Pep 703 is not a fork of CPython. Sam had (very highly impressive) protype of a gil-less cpython. It clearly demonstrate the feasibility and viability of nogil. It can be a drop in replacement, with the disabled/enable-gil option. Pep 703 is about integrating it into the main cpython codebase... > If the 2007 article and the linked current FAQ seem too old to be relevant I never said that the age of the article make irrelevant . What i said is that a lot of things change in 18 years, and it might not represent the current thinking of the person today. So i was trying to charitable to guido perspective. But in the article (from 2007) that linked it's clear that from his perspective the time/effort investment is not worth the pay out. > you're not really understanding what the steering committee is saying today. > It only takes a couple of minutes to educate yourself on this topic You keep insinuating that my current view on the situation is rooted in some kind of ignorance or some kind of misunderstanding. But you still haven't really made clear and cogent arguments for your point of view...
      - nunuvit 3y ago
        GVR called it a fork two weeks ago in his justification for not accepting it [1]. If that surprises you, the backstory is summarized pretty well in the article [2] and FAQ [3]. I'm really not trying to be secretive. We can debate whatever you want, but we have to at least acknowledge CPython's stated position. [1] https://github.com/python/steering-council/issues/188#issuecomment-1575106739 https://github.com/python/steering-council/issues/188#issuec... [2] https://www.artima.com/weblogs/viewpost.jsp?thread=214235 https://www.artima.com/weblogs/viewpost.jsp?thread=214235 [3] https://docs.python.org/3/faq/library.html#can-t-we-get-rid-of-the-global-interpreter-lock https://docs.python.org/3/faq/library.html#can-t-we-get-rid-...
kortex 3y ago
> The problem is not that deep Technically yes, the data structure can be abstracted over and removed, in that sense, it isn't "deep". But the reason I say it's deep is because of the deep implications and assumptions of the GIL being there. For decades, the codebase (and extensions) has grown with the expectation that the GIL enforces a certain behavior. > That's from my view the crust of the issue, the TLDR is that GIL-less python is not a priority for the python team, so they view any trade-off/comprise as "not justifiable". Especially something when it come to added complexity to the code base or what not... Precisely. Because it's such a hard problem. If someone had a proposal to actually remove the GIL, limit the single core performance hit to say <10%, and not break extensions, I have no doubt the steering team would go for it. It would be huge PR for python, simply because of the Gil's bad rap. Turns out that multi-interpreters give you most of the benefit of multithreading with nowhere near as much complexity. > People have this reaction because having a single-threaded interpreter in 2023 is just ... embarrassing, what ever the reasons behind it. What other dynamic languages invented over 3 decade ago have natively multithreaded interpreters? Oh javascript, the (fundamentally simpler on a technical level and uses different gc techniques) language which has gotten millions of dollars in development by the biggest tech companies because it runs on nearly every browser and thus has huge incentives to optimize? If python ran in every browser, the Gil would be gone by now. Java? Again, huge amount of backing, 15 years older than python. Perl, ruby, lua, all use refcount and have single-threaded interpreters. Objective C might fit the bill for multithreaded refcount gc (I don't know, I'm asking gpt). Nim does, but it's rather modern, (2008). Swift is statically typed, which helps, and much newer (2014). I guess that leaves Erlang, which again uses different gc techniques. I guess the takeaway is refcount is probably not the best design choice for writing a modern dynamic language if you value multithreading.
- soulbadguy 3y ago
  > But the reason I say it's deep is because of the deep implications and assumptions of the GIL being there. For decades, the codebase (and extensions) has grown with the expectation that the GIL enforces a certain behavior. As i mentioned... The idea of keeping backward compact. of an implementation details which leaked into the API is kinda common. Some of the current proposal are about making the gil optional, keeping the compact for those cases... > Precisely. Because it's such a hard problem. If someone had a proposal to actually remove the GIL, limit the single core performance hit to say <10%, and not break extensions, I have no doubt the steering team would go for it. It would be huge PR for python, simply because of the Gil's bad rap. I think this might be were we disagree the most. One doesn't sit around, put a collection of constraints into the ether and expect a solution to magically appear... Obviously every one wants a perfect solution to technical problems, but those don't really exists. The true measure of the willingness of the council to accept gil is about the compromises they are willing to make to accept external solutions or the effort put into developing their own solutions. Even if the council just had put some effort into developing a suits of test cases (both performance and correctness), or try to pull some statistical information on how many C extensions rely on the gil for correct behavior, how hard would a transition for those be? can we leverage runtime tooling like valgrind and/or llvm (thread|memory)sanitizer to detect some and ease the transition. But i haven't seen any real effort, just the same list of constraints we had the last 10 years. > Turns out that multi-interpreters give you most of the benefit of multithreading with nowhere near as much complexity. I am only tangentially familiar with the sub interpreter proposal, but what i understand, i disagree. This the classic sharing memory vs message passing trade-off... Also not to forget that this does might still break C-extension which sometime relies on static variable (like the comment from the author or mysql extension mentioned). But i don't want to make this about this or that specific proposal. if the council prefer sub interpreter approach to nogil, they should say so and don't have people waste their time. > What other dynamic languages invented over 3 decade ago have natively multithreaded interpreters? Oh javascript, the (fundamentally simpler on a technical level and uses different gc techniques) language which has gotten millions of dollars in development by the biggest tech companies because it runs on nearly every browser and thus has huge incentives to optimize? If python ran in every browser, the Gil would be gone by now. > Java? Again, huge amount of backing, 15 years older than python. > Perl, ruby, lua, all use refcount and have single-threaded interpreters. Objective C might fit the bill for multithreaded refcount gc (I don't know, I'm asking gpt). Nim does, but it's rather modern, (2008). Swift is statically typed, which helps, and much newer (2014). I don't want to sound to harsh, but you go to war with the army you have, not the army you want. The question is not what the optimal solution would look like if we had optimal conditions (deeper pockets, gc vs refc, etc... etc...). Python is what python is, the python organization have the resources that they have. No doubt we could design a better solution with more money or a with a language with better semantic, but don't have neither and that shouldn't prevent the council of acting on the best solution we have right now...

nunuvit 3y ago

It only takes a couple of minutes to educate yourself on this topic. After reading it, let me know if you still think they don't want to compromise. https://www.artima.com/weblogs/viewpost.jsp?thread=214235 https://www.artima.com/weblogs/viewpost.jsp?thread=214235

soulbadguy 3y ago
> It only takes a couple of minutes to educate yourself on this topic. You can disagree without assuming that i am not familiar with the subject. The link you provided is from 2007 so i am not sure how fair it is to quote guido from then. But i am not sure the link demonstrate the point you are trying to make, from guido himself : > While it is my personal opinion, based upon the above considerations, that there isn't enough value in removing the GIL to warrant the effort. Edit : If you want a newer reference, this HN threads and the links discribe the situation well https://news.ycombinator.com/item?id=36240998 https://news.ycombinator.com/item?id=36240998 Which is the point i am making... This is not really a technical issue, the python just doesn't think it's worth the effort.
- nunuvit 3y ago
  > From a steering council perspective we effectively view a 703 threading enabled interpreter at a high level as a fork of the CPython VM. "Fork" and "effort" are different problems. It's reductive to say that effort alone is the tldr. If the 2007 article and the linked current FAQ seem too old to be relevant, you're not really understanding what the steering committee is saying today.
  - soulbadguy 3y ago
    > "Fork" and "effort" are different problems. It's reductive to say that effort alone is the tldr. It's very unclear what you mean here, what does that have to do with the problem. Pep 703 is not a fork of CPython. Sam had (very highly impressive) protype of a gil-less cpython. It clearly demonstrate the feasibility and viability of nogil. It can be a drop in replacement, with the disabled/enable-gil option. Pep 703 is about integrating it into the main cpython codebase... > If the 2007 article and the linked current FAQ seem too old to be relevant I never said that the age of the article make irrelevant . What i said is that a lot of things change in 18 years, and it might not represent the current thinking of the person today. So i was trying to charitable to guido perspective. But in the article (from 2007) that linked it's clear that from his perspective the time/effort investment is not worth the pay out. > you're not really understanding what the steering committee is saying today. > It only takes a couple of minutes to educate yourself on this topic You keep insinuating that my current view on the situation is rooted in some kind of ignorance or some kind of misunderstanding. But you still haven't really made clear and cogent arguments for your point of view...
    - nunuvit 3y ago
      GVR called it a fork two weeks ago in his justification for not accepting it [1]. If that surprises you, the backstory is summarized pretty well in the article [2] and FAQ [3]. I'm really not trying to be secretive. We can debate whatever you want, but we have to at least acknowledge CPython's stated position. [1] https://github.com/python/steering-council/issues/188#issuecomment-1575106739 https://github.com/python/steering-council/issues/188#issuec... [2] https://www.artima.com/weblogs/viewpost.jsp?thread=214235 https://www.artima.com/weblogs/viewpost.jsp?thread=214235 [3] https://docs.python.org/3/faq/library.html#can-t-we-get-rid-of-the-global-interpreter-lock https://docs.python.org/3/faq/library.html#can-t-we-get-rid-...

kortex 3y ago

> The problem is not that deep Technically yes, the data structure can be abstracted over and removed, in that sense, it isn't "deep". But the reason I say it's deep is because of the deep implications and assumptions of the GIL being there. For decades, the codebase (and extensions) has grown with the expectation that the GIL enforces a certain behavior. > That's from my view the crust of the issue, the TLDR is that GIL-less python is not a priority for the python team, so they view any trade-off/comprise as "not justifiable". Especially something when it come to added complexity to the code base or what not... Precisely. Because it's such a hard problem. If someone had a proposal to actually remove the GIL, limit the single core performance hit to say <10%, and not break extensions, I have no doubt the steering team would go for it. It would be huge PR for python, simply because of the Gil's bad rap. Turns out that multi-interpreters give you most of the benefit of multithreading with nowhere near as much complexity. > People have this reaction because having a single-threaded interpreter in 2023 is just ... embarrassing, what ever the reasons behind it. What other dynamic languages invented over 3 decade ago have natively multithreaded interpreters? Oh javascript, the (fundamentally simpler on a technical level and uses different gc techniques) language which has gotten millions of dollars in development by the biggest tech companies because it runs on nearly every browser and thus has huge incentives to optimize? If python ran in every browser, the Gil would be gone by now. Java? Again, huge amount of backing, 15 years older than python. Perl, ruby, lua, all use refcount and have single-threaded interpreters. Objective C might fit the bill for multithreaded refcount gc (I don't know, I'm asking gpt). Nim does, but it's rather modern, (2008). Swift is statically typed, which helps, and much newer (2014). I guess that leaves Erlang, which again uses different gc techniques. I guess the takeaway is refcount is probably not the best design choice for writing a modern dynamic language if you value multithreading.

soulbadguy 3y ago
> But the reason I say it's deep is because of the deep implications and assumptions of the GIL being there. For decades, the codebase (and extensions) has grown with the expectation that the GIL enforces a certain behavior. As i mentioned... The idea of keeping backward compact. of an implementation details which leaked into the API is kinda common. Some of the current proposal are about making the gil optional, keeping the compact for those cases... > Precisely. Because it's such a hard problem. If someone had a proposal to actually remove the GIL, limit the single core performance hit to say <10%, and not break extensions, I have no doubt the steering team would go for it. It would be huge PR for python, simply because of the Gil's bad rap. I think this might be were we disagree the most. One doesn't sit around, put a collection of constraints into the ether and expect a solution to magically appear... Obviously every one wants a perfect solution to technical problems, but those don't really exists. The true measure of the willingness of the council to accept gil is about the compromises they are willing to make to accept external solutions or the effort put into developing their own solutions. Even if the council just had put some effort into developing a suits of test cases (both performance and correctness), or try to pull some statistical information on how many C extensions rely on the gil for correct behavior, how hard would a transition for those be? can we leverage runtime tooling like valgrind and/or llvm (thread|memory)sanitizer to detect some and ease the transition. But i haven't seen any real effort, just the same list of constraints we had the last 10 years. > Turns out that multi-interpreters give you most of the benefit of multithreading with nowhere near as much complexity. I am only tangentially familiar with the sub interpreter proposal, but what i understand, i disagree. This the classic sharing memory vs message passing trade-off... Also not to forget that this does might still break C-extension which sometime relies on static variable (like the comment from the author or mysql extension mentioned). But i don't want to make this about this or that specific proposal. if the council prefer sub interpreter approach to nogil, they should say so and don't have people waste their time. > What other dynamic languages invented over 3 decade ago have natively multithreaded interpreters? Oh javascript, the (fundamentally simpler on a technical level and uses different gc techniques) language which has gotten millions of dollars in development by the biggest tech companies because it runs on nearly every browser and thus has huge incentives to optimize? If python ran in every browser, the Gil would be gone by now. > Java? Again, huge amount of backing, 15 years older than python. > Perl, ruby, lua, all use refcount and have single-threaded interpreters. Objective C might fit the bill for multithreaded refcount gc (I don't know, I'm asking gpt). Nim does, but it's rather modern, (2008). Swift is statically typed, which helps, and much newer (2014). I don't want to sound to harsh, but you go to war with the army you have, not the army you want. The question is not what the optimal solution would look like if we had optimal conditions (deeper pockets, gc vs refc, etc... etc...). Python is what python is, the python organization have the resources that they have. No doubt we could design a better solution with more money or a with a language with better semantic, but don't have neither and that shouldn't prevent the council of acting on the best solution we have right now...