5 ms·

Python would be too dynamic a language to have non-GIL compatibility declared simply at top. Code can be imported/evaled/generated at runtime, and at any part o

Python would be too dynamic a language to have non-GIL compatibility declared simply at top. Code can be imported/evaled/generated at runtime, and at any part of a script, which means that python would need to be able to switch from non-GIL to GIL at any time of execution.

jedberg 3y ago
It's true that there are dynamic imports, but presumably it would be on the library maintainer to know about that, but also you could throw a catchable error about GIL imports or something like that. All I'm saying is that it's solvable, and more solvable than 2 to 3.
- samwillis 3y ago
  I completely agree, something like: from __future__ import multicore (The proposal seems to be to call it that rather than "no Gil" as it's more positive) And maybe if done in an __init__.py it applies to the whole module. Do that for v3.x then drop it completely for v4 I think the issue may be that maintaining both systems is too complex.
richdougherty 3y ago
I mean that seems horribly tricky, but also totally doable. Having read about JavaScript/Java VM optimisations in JITs and GC I would be surprised if a global state change like this is not manageable - think deoptimising the JS when you enter the debugger in the dev tools in your browser.
blibble 3y ago
Java-the-bytecode is as dynamic yet somehow hotspot manages to be rather nippy
- throwaway2037 3y ago
  This is wildly misunderstood by many just how insanely good is the current JVM. It scares me how much it has improved in the last ten years. Since it has been open source for a long time, many university researchers have done experiments and proposed HotSpot improvements. Plus, Sun/Oracle employs a small army of PhD researchers to work on it "night and day". If Project Valhalla ever sets sail (pun intended), I could imagine other languages begin to target JVM instead of their own (worse) VM. (Raku talked about it a few years ago.) When I need to move from Java to Python, I am always painfully reminded how terrible is the VM performance in a tight loop that does not use list comprehension.
  - superq 3y ago
    I'm no fan of Python after the 2-3 debacle, but to be fair, you're comparing apples and oranges here. Why not instead compare it to another C-like language (Java was originally intended to be the replacement for C++ back in 1995, and Python dates back to 1989.) Go or Rust, or even Erlang or Haskell would be great examples that are more inline with the design goals (then and now) of Java.
    - throwaway2037 3y ago
      comparing apples and oranges That is exactly my point. People start a small project in Python. Velocity is very high. Then, the project grows into a monstrosity. Suddenly, the awful Python "VM" (slow as hello) and lack of true multi-threading are major project liabilities. They should have started with someone much less sexy: DotNet (C#) or Java. Repetitive? Hell yes, but easy to maintain and cheap to hire devs.
  - lizmat 3y ago
    FWIW, the JVM is one of the supported backends of the Raku Programming Language. Has been for quite some time actually. It was the first additional backend supported after Parrot, and provided a template for supporting the MoarVM and Javascript backends since then.
phire 3y ago
That's not an insurmountable problem. As long as all the data structures stay the same, all you really need to do is flush out all the non-GIL bytecode and load in (or generate) GIL bytecode. Sure, there might be a stutter when this happens. You will also want a way to either start in GIL mode, or force it to stay in non-GIL mode, throwing errors. But it's a very solvable problem.
- Karellen 3y ago
  > all you really need to do is [SMOP]. But it's a very solvable problem. Awesome! I look forward to seeing your PR shortly. ;-)
  - throwaway2037 3y ago
    I too enjoyed the parent comment. It's right up there with "as soon as we have sufficiently smart enough compilers, that won't be an issue anymore". I needed to Google SMOP, which tells me: "small matter of programming". Correct?
    - phire 3y ago
      I'm confused. Why do people think my suggested solution is difficult enough to be comparable to the halting problem or "a smart enough compiler"? We are talking about a scenario where cpython has already been modified to support both GIL and non-GIL operating modes, and a set of libraries that support the new non-GIL mode have been manually annotated by programmers. All I'm suggesting is that on top of this, we modify the "load library" code to detect at runtime when we are loading a library that doesn't support the new non-GIL mode and switch modes.
      - throwaway2037 3y ago
        This simplest solution is to get a patch accepted by Python team. Good luck with that.
    - Karellen 3y ago

jedberg 3y ago

It's true that there are dynamic imports, but presumably it would be on the library maintainer to know about that, but also you could throw a catchable error about GIL imports or something like that. All I'm saying is that it's solvable, and more solvable than 2 to 3.

samwillis 3y ago
I completely agree, something like: from __future__ import multicore (The proposal seems to be to call it that rather than "no Gil" as it's more positive) And maybe if done in an __init__.py it applies to the whole module. Do that for v3.x then drop it completely for v4 I think the issue may be that maintaining both systems is too complex.

richdougherty 3y ago

I mean that seems horribly tricky, but also totally doable. Having read about JavaScript/Java VM optimisations in JITs and GC I would be surprised if a global state change like this is not manageable - think deoptimising the JS when you enter the debugger in the dev tools in your browser.

blibble 3y ago

Java-the-bytecode is as dynamic yet somehow hotspot manages to be rather nippy

throwaway2037 3y ago
This is wildly misunderstood by many just how insanely good is the current JVM. It scares me how much it has improved in the last ten years. Since it has been open source for a long time, many university researchers have done experiments and proposed HotSpot improvements. Plus, Sun/Oracle employs a small army of PhD researchers to work on it "night and day". If Project Valhalla ever sets sail (pun intended), I could imagine other languages begin to target JVM instead of their own (worse) VM. (Raku talked about it a few years ago.) When I need to move from Java to Python, I am always painfully reminded how terrible is the VM performance in a tight loop that does not use list comprehension.
- superq 3y ago
  I'm no fan of Python after the 2-3 debacle, but to be fair, you're comparing apples and oranges here. Why not instead compare it to another C-like language (Java was originally intended to be the replacement for C++ back in 1995, and Python dates back to 1989.) Go or Rust, or even Erlang or Haskell would be great examples that are more inline with the design goals (then and now) of Java.
  - throwaway2037 3y ago
    comparing apples and oranges That is exactly my point. People start a small project in Python. Velocity is very high. Then, the project grows into a monstrosity. Suddenly, the awful Python "VM" (slow as hello) and lack of true multi-threading are major project liabilities. They should have started with someone much less sexy: DotNet (C#) or Java. Repetitive? Hell yes, but easy to maintain and cheap to hire devs.
- lizmat 3y ago
  FWIW, the JVM is one of the supported backends of the Raku Programming Language. Has been for quite some time actually. It was the first additional backend supported after Parrot, and provided a template for supporting the MoarVM and Javascript backends since then.

phire 3y ago

That's not an insurmountable problem. As long as all the data structures stay the same, all you really need to do is flush out all the non-GIL bytecode and load in (or generate) GIL bytecode. Sure, there might be a stutter when this happens. You will also want a way to either start in GIL mode, or force it to stay in non-GIL mode, throwing errors. But it's a very solvable problem.

Karellen 3y ago
> all you really need to do is [SMOP]. But it's a very solvable problem. Awesome! I look forward to seeing your PR shortly. ;-)
- throwaway2037 3y ago
  I too enjoyed the parent comment. It's right up there with "as soon as we have sufficiently smart enough compilers, that won't be an issue anymore". I needed to Google SMOP, which tells me: "small matter of programming". Correct?
  - phire 3y ago
    I'm confused. Why do people think my suggested solution is difficult enough to be comparable to the halting problem or "a smart enough compiler"? We are talking about a scenario where cpython has already been modified to support both GIL and non-GIL operating modes, and a set of libraries that support the new non-GIL mode have been manually annotated by programmers. All I'm suggesting is that on top of this, we modify the "load library" code to detect at runtime when we are loading a library that doesn't support the new non-GIL mode and switch modes.
    - throwaway2037 3y ago
      This simplest solution is to get a patch accepted by Python team. Good luck with that.
  - Karellen 3y ago