7 ms·

Why? Java has no GIL and runs faster than Python.

chrisseaton 5y ago
Java has different semantics to Python. To make Python run fast without a GIL you'd have to change the semantics to be more like Java's, which would be a breaking change, which would mean Python 4.0, which is what they say we're unlikely to get.
- dekhn 5y ago
  IronPython, based on CLR and .NET, was faster than Python and didn't have a GIL.
  - deleted 5y ago
    [deleted]
  - pansa2 5y ago
    IIRC it included a JIT compiler, which was fast enough to offset the overhead of fine-granularity locking.
    - zerkten 5y ago
      You have to consider the use case in addition to the technology. The relatively simple implementation of CPython ensures you have fast startup, but trade-off other optimizations which affect long-running programs. Lots of Python scripts only run for a short time. Code in the REPL needs to return results really quickly. The .NET JIT added significant startup costs to the point where the IronPython team had to add an interpreted mode for first runs before handing off the JIT. This is complex and arguably takes time away from CPython compatibility work. It seems that the .NET team is coming around to the concerns around startup time, but I don't know if they have landed on an AOT solution. It probably will help executables running on .NET Core more than scripts running under IronPython. I've not kept up with JITs very much, but I don't think this pattern is limited to the .NET JIT, and is a general trade-off. I feel like JavaScript JITs may be closer to what IronPython would have wanted though.
  - dagw 5y ago
    IronPython also wasn't compatible with any python library that used the C API. Essentially we're back to the point that removing the GIL is 'easy' as long as you don't care about backwards compatibility.
    - dekhn 5y ago
      in many ways ironpython obviated the C API. It had a different approach to extension, based on CLR and .NET. Even the containers used .NET data types.
      - dagw 5y ago
        Oh absolutely, but that did make IronPython more of a separate language that shared syntax with cpython rather than a drop in replacement for cpython.
  - chrisseaton 5y ago
    But, again, IronPython does not have the same semantics as standard Python, so this would be a breaking change to adopt, and so would need to be Python 4.
- needusername 5y ago
  Why? Which different semantics cause a problem?
  - chrisseaton 5y ago
    Currently Python only allows certain thread interleavings, because those are the points where the GIL is released. If you no longer had the GIL, the set of possible thread interleavings would increase. If your application is not safe with a particular interleaving, it can now break. That's a breaking change, so would require Python 4. It might be possible to retain current interleaving semantics, through research-level techniques such as transactional memory, but these do not seem likely anytime soon despite a lot of work. It's also unclear how tractable it is to retain current garbage collection semantics without a GIL, so that could have to change as well with similar issues and similar possible solutions which seem unlikely to land anytime soon.
    - needusername 5y ago
      > Currently Python only allows certain thread interleavings, because those are the points where the GIL is released. How is this currently avoided? My understanding is that anything that accepts a callback / lambda is potentially subject to interleavings as the code can call a C extension which releases the GIL. Moving code to C extensions is often recommended when Python performance is brought up. Am I misunderstanding something? > It's also unclear how tractable it is to retain current garbage collection semantics without a GIL Why is it important to retain current garbage collection semantics? Why does Python prescribe a certain garbage collection implementation and does not allow for others like Java?
  - anamax 5y ago
    The big difference is that Python doesn't allow very much static analysis. Consider the expression "a.b.c". Where are the function calls? In other words, which of the ".{name}" return a value from some namespace and which call code? (I'm not asking about which functions are called. I'm just asking about where functions are called.) Java semantics, including the type system and the required declarations, tell you where the function calls are. One consequence is that they're always in the same place. That's simply not possible in Python because the answer isn't known until the expression is evaluated AND the answer can change every time the expression is evaluated.
pansa2 5y ago
AFAIK multithreaded Java code requires the user to explicitly acquire a lock before accessing a shared data structure. Multithreaded Python currently does not, and must remain that way to maintain compatibility.
- needusername 5y ago
  > Multithreaded Python currently does not How do you prevent interleaving of different threads then? > and must remain that way to maintain compatibility. Why? Python 3 broke a whole lot of applications and libraries.
  - pansa2 5y ago
    > How do you prevent interleaving of different threads then? The GIL. > Why? Python 3 broke a whole lot of applications and libraries. Sorry, I mean it must remain that way if it is to maintain compatibility. We’re assuming that breaking compatibility is unacceptable.
    - needusername 5y ago
      > The GIL. How does the the GIL prevent interleaving? My understanding is that if an "operation" takes several steps, eg. reading from a dict then IO or a C extension and finally writing to a dict the GIL will not make this atomic.
    - formerly_proven 5y ago
      Most Python 3 breakage was search-replace fixable (though I'd question the value of language changes falling in this category), although it is true that the text/bytes separation exposed unsound data models in some applications, which is of course nasty to deal with.
      - needusername 5y ago
        Why did it take projects and Google so long to migrate?

Java has different semantics to Python. To make Python run fast without a GIL you'd have to change the semantics to be more like Java's, which would be a breaking change, which would mean Python 4.0, which is what they say we're unlikely to get.

dekhn 5y ago
IronPython, based on CLR and .NET, was faster than Python and didn't have a GIL.
- deleted 5y ago
  [deleted]
- pansa2 5y ago
  IIRC it included a JIT compiler, which was fast enough to offset the overhead of fine-granularity locking.
  - zerkten 5y ago
    You have to consider the use case in addition to the technology. The relatively simple implementation of CPython ensures you have fast startup, but trade-off other optimizations which affect long-running programs. Lots of Python scripts only run for a short time. Code in the REPL needs to return results really quickly. The .NET JIT added significant startup costs to the point where the IronPython team had to add an interpreted mode for first runs before handing off the JIT. This is complex and arguably takes time away from CPython compatibility work. It seems that the .NET team is coming around to the concerns around startup time, but I don't know if they have landed on an AOT solution. It probably will help executables running on .NET Core more than scripts running under IronPython. I've not kept up with JITs very much, but I don't think this pattern is limited to the .NET JIT, and is a general trade-off. I feel like JavaScript JITs may be closer to what IronPython would have wanted though.
- dagw 5y ago
  IronPython also wasn't compatible with any python library that used the C API. Essentially we're back to the point that removing the GIL is 'easy' as long as you don't care about backwards compatibility.
  - dekhn 5y ago
    in many ways ironpython obviated the C API. It had a different approach to extension, based on CLR and .NET. Even the containers used .NET data types.
    - dagw 5y ago
      Oh absolutely, but that did make IronPython more of a separate language that shared syntax with cpython rather than a drop in replacement for cpython.
- chrisseaton 5y ago
  But, again, IronPython does not have the same semantics as standard Python, so this would be a breaking change to adopt, and so would need to be Python 4.
needusername 5y ago
Why? Which different semantics cause a problem?
- chrisseaton 5y ago
  Currently Python only allows certain thread interleavings, because those are the points where the GIL is released. If you no longer had the GIL, the set of possible thread interleavings would increase. If your application is not safe with a particular interleaving, it can now break. That's a breaking change, so would require Python 4. It might be possible to retain current interleaving semantics, through research-level techniques such as transactional memory, but these do not seem likely anytime soon despite a lot of work. It's also unclear how tractable it is to retain current garbage collection semantics without a GIL, so that could have to change as well with similar issues and similar possible solutions which seem unlikely to land anytime soon.
  - needusername 5y ago
    > Currently Python only allows certain thread interleavings, because those are the points where the GIL is released. How is this currently avoided? My understanding is that anything that accepts a callback / lambda is potentially subject to interleavings as the code can call a C extension which releases the GIL. Moving code to C extensions is often recommended when Python performance is brought up. Am I misunderstanding something? > It's also unclear how tractable it is to retain current garbage collection semantics without a GIL Why is it important to retain current garbage collection semantics? Why does Python prescribe a certain garbage collection implementation and does not allow for others like Java?
- anamax 5y ago
  The big difference is that Python doesn't allow very much static analysis. Consider the expression "a.b.c". Where are the function calls? In other words, which of the ".{name}" return a value from some namespace and which call code? (I'm not asking about which functions are called. I'm just asking about where functions are called.) Java semantics, including the type system and the required declarations, tell you where the function calls are. One consequence is that they're always in the same place. That's simply not possible in Python because the answer isn't known until the expression is evaluated AND the answer can change every time the expression is evaluated.

pansa2 5y ago

AFAIK multithreaded Java code requires the user to explicitly acquire a lock before accessing a shared data structure. Multithreaded Python currently does not, and must remain that way to maintain compatibility.

needusername 5y ago
> Multithreaded Python currently does not How do you prevent interleaving of different threads then? > and must remain that way to maintain compatibility. Why? Python 3 broke a whole lot of applications and libraries.
- pansa2 5y ago
  > How do you prevent interleaving of different threads then? The GIL. > Why? Python 3 broke a whole lot of applications and libraries. Sorry, I mean it must remain that way if it is to maintain compatibility. We’re assuming that breaking compatibility is unacceptable.
  - needusername 5y ago
    > The GIL. How does the the GIL prevent interleaving? My understanding is that if an "operation" takes several steps, eg. reading from a dict then IO or a C extension and finally writing to a dict the GIL will not make this atomic.
  - formerly_proven 5y ago
    Most Python 3 breakage was search-replace fixable (though I'd question the value of language changes falling in this category), although it is true that the text/bytes separation exposed unsound data models in some applications, which is of course nasty to deal with.
    - needusername 5y ago
      Why did it take projects and Google so long to migrate?