5 ms·

> The GIL will never be removed from the main python implementation. I don't see why. It's a much easier transition than 2 to 3. Make each package declare ri

> The GIL will never be removed from the main python implementation. I don't see why. It's a much easier transition than 2 to 3. Make each package declare right at the top if they are non-GIL compatible. Have both modes available in the interpreter. If every piece of code imported has the declaration on top, then it runs in non-GIL mode, otherwise it runs in classic GIL mode. At first most code would still run in GIL mode, but over time most packages would be converted, especially if people stopped using packages that weren't compatible.

powersnail 3y ago
Python would be too dynamic a language to have non-GIL compatibility declared simply at top. Code can be imported/evaled/generated at runtime, and at any part of a script, which means that python would need to be able to switch from non-GIL to GIL at any time of execution.
- jedberg 3y ago
  It's true that there are dynamic imports, but presumably it would be on the library maintainer to know about that, but also you could throw a catchable error about GIL imports or something like that. All I'm saying is that it's solvable, and more solvable than 2 to 3.
  - samwillis 3y ago
    I completely agree, something like: from __future__ import multicore (The proposal seems to be to call it that rather than "no Gil" as it's more positive) And maybe if done in an __init__.py it applies to the whole module. Do that for v3.x then drop it completely for v4 I think the issue may be that maintaining both systems is too complex.
- richdougherty 3y ago
  I mean that seems horribly tricky, but also totally doable. Having read about JavaScript/Java VM optimisations in JITs and GC I would be surprised if a global state change like this is not manageable - think deoptimising the JS when you enter the debugger in the dev tools in your browser.
- blibble 3y ago
  Java-the-bytecode is as dynamic yet somehow hotspot manages to be rather nippy
  - throwaway2037 3y ago
    This is wildly misunderstood by many just how insanely good is the current JVM. It scares me how much it has improved in the last ten years. Since it has been open source for a long time, many university researchers have done experiments and proposed HotSpot improvements. Plus, Sun/Oracle employs a small army of PhD researchers to work on it "night and day". If Project Valhalla ever sets sail (pun intended), I could imagine other languages begin to target JVM instead of their own (worse) VM. (Raku talked about it a few years ago.) When I need to move from Java to Python, I am always painfully reminded how terrible is the VM performance in a tight loop that does not use list comprehension.
    - superq 3y ago
      I'm no fan of Python after the 2-3 debacle, but to be fair, you're comparing apples and oranges here. Why not instead compare it to another C-like language (Java was originally intended to be the replacement for C++ back in 1995, and Python dates back to 1989.) Go or Rust, or even Erlang or Haskell would be great examples that are more inline with the design goals (then and now) of Java.
      - throwaway2037 3y ago
        comparing apples and oranges That is exactly my point. People start a small project in Python. Velocity is very high. Then, the project grows into a monstrosity. Suddenly, the awful Python "VM" (slow as hello) and lack of true multi-threading are major project liabilities. They should have started with someone much less sexy: DotNet (C#) or Java. Repetitive? Hell yes, but easy to maintain and cheap to hire devs.
    - lizmat 3y ago
      FWIW, the JVM is one of the supported backends of the Raku Programming Language. Has been for quite some time actually. It was the first additional backend supported after Parrot, and provided a template for supporting the MoarVM and Javascript backends since then.
- phire 3y ago
  That's not an insurmountable problem. As long as all the data structures stay the same, all you really need to do is flush out all the non-GIL bytecode and load in (or generate) GIL bytecode. Sure, there might be a stutter when this happens. You will also want a way to either start in GIL mode, or force it to stay in non-GIL mode, throwing errors. But it's a very solvable problem.
  - Karellen 3y ago
    > all you really need to do is [SMOP]. But it's a very solvable problem. Awesome! I look forward to seeing your PR shortly. ;-)
    - throwaway2037 3y ago
      I too enjoyed the parent comment. It's right up there with "as soon as we have sufficiently smart enough compilers, that won't be an issue anymore". I needed to Google SMOP, which tells me: "small matter of programming". Correct?
      - phire 3y ago
        I'm confused. Why do people think my suggested solution is difficult enough to be comparable to the halting problem or "a smart enough compiler"? We are talking about a scenario where cpython has already been modified to support both GIL and non-GIL operating modes, and a set of libraries that support the new non-GIL mode have been manually annotated by programmers. All I'm suggesting is that on top of this, we modify the "load library" code to detect at runtime when we are loading a library that doesn't support the new non-GIL mode and switch modes.
        throwaway2037 3y ago
        This simplest solution is to get a patch accepted by Python team. Good luck with that.
      - Karellen 3y ago
dekhn 3y ago
I'm sorry but I really don't think it's "easy" and your suggestion would be just part of a much, much larger solution.
aprdm 3y ago
[flagged]
- xapata 3y ago
  jedberg doesn't seem like a naïve person, glancing at his work history. Perhaps you could explain why you think this opinion is naïve?
  - Eisenstein 3y ago
    "There are two types of programmers: new ones who don't know how complicated things are, and experienced ones who have forgotten it" -from an article on HN the other day about setting up python envs.
    - execveat 3y ago
      There also are programmers who are aware of much more complicated things being done in sister projects, like JVM and JS runtimes in this case.
    - xapata 3y ago
      An appeal to authority? I suppose it'd be better if you named the article's author or linked to it.
      - Eisenstein 3y ago
        Not appealing to anyone; I thought it was a clever quote. * https://www.bitecode.dev/p/why-not-tell-people-to-simply-use https://www.bitecode.dev/p/why-not-tell-people-to-simply-use EDIT -- Here is the quote: There are usually two kinds of coders giving advises. A fresh one that has no idea how complex things really are, yet. Or an experienced one, that forgot it.
        xapata 3y ago
        It's clever, but like (almost) all such things, doesn't always apply.
Groxx 3y ago
I don't see why you wouldn't have holdouts using GIL for valid single-threaded performance reasons for years/decades. And that's ignoring legacy code - even 2.7 is still alive and kicking in some corners.
- jedberg 3y ago
  I'm sure you would, but anyone who cared about it would work around it, either by not using that library or finding a different one or even forking the one that isn't updated. Just like most people use Python 3.x now, but there are some 2.7 holdouts. But those holdouts aren't holding back the entire ecosystem at this point.
  - Groxx 3y ago
    They did for many, many years. And the result soured many devs and companies on Python. And many libraries have not been migrated. But yes, I'm very glad they've been cut off now. It does eventually shift to just the new stuff.
    - zarzavat 3y ago
      The difference is that no-GIL is a minority use case. Most people are using Python in single threaded workflows (numpy/scipy, machine learning, build scripts, etc etc) and who don’t care about the GIL. The GIL mostly affects webdevs. Probably the library support will be much worse than 2-3, but if only a minority of users are impacted by libraries that don’t support it then it’s not a big issue. GIL concerned users can either carefully select their dependencies to be no-GIL supporting, or they can decide to go with the GIL. 2-3 was bad because every single library, app, and tiny script had to be migrated, if it used a string or a print statement.

powersnail 3y ago

Python would be too dynamic a language to have non-GIL compatibility declared simply at top. Code can be imported/evaled/generated at runtime, and at any part of a script, which means that python would need to be able to switch from non-GIL to GIL at any time of execution.

jedberg 3y ago
It's true that there are dynamic imports, but presumably it would be on the library maintainer to know about that, but also you could throw a catchable error about GIL imports or something like that. All I'm saying is that it's solvable, and more solvable than 2 to 3.
- samwillis 3y ago
  I completely agree, something like: from __future__ import multicore (The proposal seems to be to call it that rather than "no Gil" as it's more positive) And maybe if done in an __init__.py it applies to the whole module. Do that for v3.x then drop it completely for v4 I think the issue may be that maintaining both systems is too complex.
richdougherty 3y ago
I mean that seems horribly tricky, but also totally doable. Having read about JavaScript/Java VM optimisations in JITs and GC I would be surprised if a global state change like this is not manageable - think deoptimising the JS when you enter the debugger in the dev tools in your browser.
blibble 3y ago
Java-the-bytecode is as dynamic yet somehow hotspot manages to be rather nippy
- throwaway2037 3y ago
  This is wildly misunderstood by many just how insanely good is the current JVM. It scares me how much it has improved in the last ten years. Since it has been open source for a long time, many university researchers have done experiments and proposed HotSpot improvements. Plus, Sun/Oracle employs a small army of PhD researchers to work on it "night and day". If Project Valhalla ever sets sail (pun intended), I could imagine other languages begin to target JVM instead of their own (worse) VM. (Raku talked about it a few years ago.) When I need to move from Java to Python, I am always painfully reminded how terrible is the VM performance in a tight loop that does not use list comprehension.
  - superq 3y ago
    I'm no fan of Python after the 2-3 debacle, but to be fair, you're comparing apples and oranges here. Why not instead compare it to another C-like language (Java was originally intended to be the replacement for C++ back in 1995, and Python dates back to 1989.) Go or Rust, or even Erlang or Haskell would be great examples that are more inline with the design goals (then and now) of Java.
    - throwaway2037 3y ago
      comparing apples and oranges That is exactly my point. People start a small project in Python. Velocity is very high. Then, the project grows into a monstrosity. Suddenly, the awful Python "VM" (slow as hello) and lack of true multi-threading are major project liabilities. They should have started with someone much less sexy: DotNet (C#) or Java. Repetitive? Hell yes, but easy to maintain and cheap to hire devs.
  - lizmat 3y ago
    FWIW, the JVM is one of the supported backends of the Raku Programming Language. Has been for quite some time actually. It was the first additional backend supported after Parrot, and provided a template for supporting the MoarVM and Javascript backends since then.
phire 3y ago
That's not an insurmountable problem. As long as all the data structures stay the same, all you really need to do is flush out all the non-GIL bytecode and load in (or generate) GIL bytecode. Sure, there might be a stutter when this happens. You will also want a way to either start in GIL mode, or force it to stay in non-GIL mode, throwing errors. But it's a very solvable problem.
- Karellen 3y ago
  > all you really need to do is [SMOP]. But it's a very solvable problem. Awesome! I look forward to seeing your PR shortly. ;-)
  - throwaway2037 3y ago
    I too enjoyed the parent comment. It's right up there with "as soon as we have sufficiently smart enough compilers, that won't be an issue anymore". I needed to Google SMOP, which tells me: "small matter of programming". Correct?
    - phire 3y ago
      I'm confused. Why do people think my suggested solution is difficult enough to be comparable to the halting problem or "a smart enough compiler"? We are talking about a scenario where cpython has already been modified to support both GIL and non-GIL operating modes, and a set of libraries that support the new non-GIL mode have been manually annotated by programmers. All I'm suggesting is that on top of this, we modify the "load library" code to detect at runtime when we are loading a library that doesn't support the new non-GIL mode and switch modes.
      - throwaway2037 3y ago
        This simplest solution is to get a patch accepted by Python team. Good luck with that.
    - Karellen 3y ago

dekhn 3y ago

I'm sorry but I really don't think it's "easy" and your suggestion would be just part of a much, much larger solution.

aprdm 3y ago

[flagged]

xapata 3y ago
jedberg doesn't seem like a naïve person, glancing at his work history. Perhaps you could explain why you think this opinion is naïve?
- Eisenstein 3y ago
  "There are two types of programmers: new ones who don't know how complicated things are, and experienced ones who have forgotten it" -from an article on HN the other day about setting up python envs.
  - execveat 3y ago
    There also are programmers who are aware of much more complicated things being done in sister projects, like JVM and JS runtimes in this case.
  - xapata 3y ago
    An appeal to authority? I suppose it'd be better if you named the article's author or linked to it.
    - Eisenstein 3y ago
      Not appealing to anyone; I thought it was a clever quote. * https://www.bitecode.dev/p/why-not-tell-people-to-simply-use https://www.bitecode.dev/p/why-not-tell-people-to-simply-use EDIT -- Here is the quote: There are usually two kinds of coders giving advises. A fresh one that has no idea how complex things really are, yet. Or an experienced one, that forgot it.
      - xapata 3y ago
        It's clever, but like (almost) all such things, doesn't always apply.

Groxx 3y ago

I don't see why you wouldn't have holdouts using GIL for valid single-threaded performance reasons for years/decades. And that's ignoring legacy code - even 2.7 is still alive and kicking in some corners.

jedberg 3y ago
I'm sure you would, but anyone who cared about it would work around it, either by not using that library or finding a different one or even forking the one that isn't updated. Just like most people use Python 3.x now, but there are some 2.7 holdouts. But those holdouts aren't holding back the entire ecosystem at this point.
- Groxx 3y ago
  They did for many, many years. And the result soured many devs and companies on Python. And many libraries have not been migrated. But yes, I'm very glad they've been cut off now. It does eventually shift to just the new stuff.
  - zarzavat 3y ago
    The difference is that no-GIL is a minority use case. Most people are using Python in single threaded workflows (numpy/scipy, machine learning, build scripts, etc etc) and who don’t care about the GIL. The GIL mostly affects webdevs. Probably the library support will be much worse than 2-3, but if only a minority of users are impacted by libraries that don’t support it then it’s not a big issue. GIL concerned users can either carefully select their dependencies to be no-GIL supporting, or they can decide to go with the GIL. 2-3 was bad because every single library, app, and tiny script had to be migrated, if it used a string or a print statement.