6 ms·

From the write-up, I honestly don't understand how this paves the way. I don't see an architectural path from a cut-and-paste JIT to something optimizing. That'

by blagie 3y ago

From the write-up, I honestly don't understand how this paves the way. I don't see an architectural path from a cut-and-paste JIT to something optimizing. That's the whole point of a cut-and-paste JIT.

fulafel 3y ago
Support for generating machine code at all seems like a necessary building block to me and probably is quite a bit of effort to work on top of a portable interpreter code base.
lumpa 3y ago
There's a lot of effort going on to improve CPython performance, with optimization tiers, etc. It seems the JIT is how at least part of that effort will materialize: https://github.com/python/cpython/issues/113710 https://github.com/python/cpython/issues/113710 > We're getting a JIT. Now it's time to optimize the traces to pass them to the JIT.
guenthert 3y ago
Isn't it the case that Python allows for type specifier (type hints) since 3.5, albeit the CPython interpreter ignores them? The JIT might take advantage of them, which ought to improve performance significantly for some code. That what makes Python flexible is what makes it slow. Restricting the flexibility were possible offers opportunities to improve performance (and allows for tools and humans to spot errors more easily).
- a-french-anon 3y ago
  Isn't CL a good counter-example to that "dynamism inherently stunts performances" mantra?
  - guenthert 3y ago
    To the contrary. In CL some flexibility was given up (compared to other LISP dialects) in favor of enabling optimizing compilers, e.g. the standard symbols cannot be reassigned (also preserving the sanity of human readers). CL also offers what some now call 'gradual typing', i.e. optional type declarations. And remaining flexibility, e.g. around the OO support, limits how well the compiler can optimize the code.
    - Joker_vD 3y ago
      But type declarations in Python are not required to be correct, are they? You are allowed to write def twice(x: int) -> int: return x + x print(twice("nope")) and it should print "nopenope". Right?
      - Ringz 3y ago
        Yep. Therefore it’s better to def twice(x: int) -> int: if not isinstance(x, int): raise TypeError("Expected x to be an int, got " + str(type(x))) return x + x
        jhardy54 3y ago
        This can have substantial performance implications, not to mention DX considerations.
        Ringz 3y ago
        Of course, this is not a good example of good, high-performance code, only an answer to the specific question... the questioner certainly also knows MyPy.
        Joker_vD 3y ago
        I actually don't know anything about MyPy, only that it exists. Does it run that example correctly, that is, does it print "nopenope"? Because I think it's the correct behaviour, type hints should not actually affect evaluation (well, beyond the fact that they must be names that are visible in the scopes thay're used in, obviously), altough I could be wrong. Besides, my point was that one of the reasons why languages with (sound-ish) static types manage to have better performance because they can omit all of those run-time type checks (and the supporting machinery) because they'd never fail. And if you have to put those explicit checks, then the type hints are actually entirely redundant: e.g. Erlang's JIT ignores type specs, it instead looks at the type guards in the code to generate specialized code for the function bodies.
- formerly_proven 3y ago
  You can't really on type annotations to help interpret the code.
- Difwif 3y ago
  AFAIK good JITs like V8 can do runtime introspection and recompile on the fly if types change. Maybe using the type hints will be helpful but I don't think they are necessary for significant improvement.
  - amelius 3y ago
    Are there any benchmarks that give an idea of how much this might improve Python's speed?
    - mike_hearn 3y ago
      Well, GraalPython is a Python JIT compiler which can exploit dynamically determined types, and it advertises 4.3x faster, so it's possible to do drastically better than a few percent. I think that's state of the art but might be wrong. That's for this benchmark: https://pyperformance.readthedocs.io/ https://pyperformance.readthedocs.io/ Note that this is with a relatively small investment as these things go, the GraalPython team is about ~3 people I guess, looking at the GH repo. It's an independent implementation so most of the work went into being compatible with Python including native extensions (the hard part). But this speedup depends a lot on what you're doing. Some types of code can go much faster. Others will be slower even than CPython, for example if you want to sandbox the native code extensions.
      - amelius 3y ago
        This is great info, thanks!
    - pletnes 3y ago
      Pypy is a different JIT that gives anything from slower/same to 100x speedup depending on the benchmark. They give a geometric mean of 4.8x speedup across their suite of benchmarks. https://speed.pypy.org/ https://speed.pypy.org/
  - cuchoi 3y ago
    Doesn't Python already do this? https://www.youtube.com/watch?v=shQtrn1v7sQ https://www.youtube.com/watch?v=shQtrn1v7sQ
- dataangel 3y ago
  I doubt it with a copy-and-patch JIT, not the way they work now. I'm a serious mypy/python-static-types user and as is they currently wouldn't allow you to do much optimization wise. - All integers are still big integers - Use of the typing opt-out 'Any' is very common - All functions/methods can still be overwritten at runtime - Fields can still be added and removed from objects at runtime The combination basically makes it mandatory to not use native arithmetic, allocate everything on the heap, and need multiple levels of indirection for looking up any variable/field/function. CPU perf nightmare. You need a real optimizing JIT to track when integers are in a narrow range and things aren't getting redefined at runtime.
- tekknolagi 3y ago
  Sort of! But also not really. If you want to get into this, I wrote a post about this: https://bernsteinbear.com/blog/typed-python/ https://bernsteinbear.com/blog/typed-python/
Someone 3y ago
It should be fairly easy to add instruction fusing, where they recognize often-used instruction pairs, combine their C code, and then let the compiler optimize the combined code. Combining LOAD_CONST with the instruction following it if that instruction pops the const from the stack seems an easy win, for example.
- ncruces 3y ago
  If it was that easy, you'd do that in the interpreter and proportionally reduce interpretation overhead.
  - Someone 3y ago
    In the interpreter, I don’t think it wouldn’t reduce overhead much, if at all. You’d still have to recognize the two byte codes, and your interpreter would spend additional time deciding, for most byte code pairs, that it doesn’t know how to combine them. With a compiler, that part is done once and, potentially, run zillions of times.
    - ncruces 3y ago
      If fusing a certain pair would significantly improve performance of most code, you'd just add that fused instruction to your bytecode and let the C compiler optimize the combined code in the interpreter. I have to assume CPython as already done that for all the low hanging fruit. In fact, for such a fused instruction to be optimized that way on a copy-and-patch JIT it'd need to exist as a new bytecode in interpreter. A JIT that fuses instructions is no longer a copy-and-patch JIT. A copy-and-patch JIT reduces interpretation overhead by making sure the branches in the executed machine code are the branches in the code to be interpreted, not branches in the interpreter. This is make a huge difference in more naive interpreters, not so much in an heavily optimized threaded-code interpreter. The 10% is great, and nothing to sneeze at for a first commit. But I'd actually like some realistic analysis of next steps for improvement, because I'm skeptical instruction fusing and other things being hand waved are it. Certainly not on a copy-and-patch JIT. For context: I spent significant effort trying to add such instruction fusing to a simple WASM AOT compiler and got nowhere (the equivalent of constant loading was precisely one of the pairs). Only moving to a much smarter JIT (capable of looking at whole basic blocks of instructions) started making a difference.
londons_explore 3y ago
> . I don't see an architectural path from a cut-and-paste JIT to something optimizing. One approach used in V8 is to have a dumb-but-very-fast JIT (ie. this), and keep counters of how often each block of code runs (perhaps actual counters, perhaps using CPU sampling features), and then any block of code running more than a few thousand times run through a far more complex yet slower optimizing jit. That has the benefit that the 0.2% of your code which uses 95% of the runtime is the only part that has to undergo the expensive optimization passes.
- Sesse__ 3y ago
  Note that V8 didn't have a dumb-but-very-fast JIT (Sparkplug) until 2021; the interpreter (Ignition) did that block counting and sent it straight to the optimizing JIT (TurboFan). V8 pre-2021 (i.e., only Ignition+TurboFan) was significantly faster than current CPython is, and the full current four-tier bundle (Ignition+Sparkplug+Maglev+TurboFan) only scores roughly twice as good on Speedometer as pure Ignition does. (Ignition+Sparkplug is about 40% faster than Ignition alone; compare that “dumbness” with CPython's 2–9%.) The relevant lesson should be that things like very carefully designed value representation and IR is a much more important piece of the puzzle than having as many tiers of compilation as possible.
  - uluyol 3y ago
    In case anyone is interested, V8 pre-ignition/TurboFan had different tiers [1]: full-codegen (dumb and fast) and crankshaft (optimizing). It's interesting to see how these things change over time. [1]: https://v8.dev/blog/ignition-interpreter https://v8.dev/blog/ignition-interpreter
- dataangel 3y ago
  > keep counters of how often each block of code runs ... and then any block of code running more than a few thousand times run through a far more complex yet slower optimizing jit. That's just all JITs. Sometimes its counters for going from interpreter -> JIT rather than levels of JITs, but this idea is as old as JITs.

by blagie 3y ago

fulafel 3y ago

Support for generating machine code at all seems like a necessary building block to me and probably is quite a bit of effort to work on top of a portable interpreter code base.

lumpa 3y ago

There's a lot of effort going on to improve CPython performance, with optimization tiers, etc. It seems the JIT is how at least part of that effort will materialize: https://github.com/python/cpython/issues/113710 https://github.com/python/cpython/issues/113710 > We're getting a JIT. Now it's time to optimize the traces to pass them to the JIT.

guenthert 3y ago

Isn't it the case that Python allows for type specifier (type hints) since 3.5, albeit the CPython interpreter ignores them? The JIT might take advantage of them, which ought to improve performance significantly for some code. That what makes Python flexible is what makes it slow. Restricting the flexibility were possible offers opportunities to improve performance (and allows for tools and humans to spot errors more easily).

a-french-anon 3y ago
Isn't CL a good counter-example to that "dynamism inherently stunts performances" mantra?
- guenthert 3y ago
  To the contrary. In CL some flexibility was given up (compared to other LISP dialects) in favor of enabling optimizing compilers, e.g. the standard symbols cannot be reassigned (also preserving the sanity of human readers). CL also offers what some now call 'gradual typing', i.e. optional type declarations. And remaining flexibility, e.g. around the OO support, limits how well the compiler can optimize the code.
  - Joker_vD 3y ago
    But type declarations in Python are not required to be correct, are they? You are allowed to write def twice(x: int) -> int: return x + x print(twice("nope")) and it should print "nopenope". Right?
    - Ringz 3y ago
      Yep. Therefore it’s better to def twice(x: int) -> int: if not isinstance(x, int): raise TypeError("Expected x to be an int, got " + str(type(x))) return x + x
      - jhardy54 3y ago
        This can have substantial performance implications, not to mention DX considerations.
        Ringz 3y ago
        Of course, this is not a good example of good, high-performance code, only an answer to the specific question... the questioner certainly also knows MyPy.
        Joker_vD 3y ago
        I actually don't know anything about MyPy, only that it exists. Does it run that example correctly, that is, does it print "nopenope"? Because I think it's the correct behaviour, type hints should not actually affect evaluation (well, beyond the fact that they must be names that are visible in the scopes thay're used in, obviously), altough I could be wrong. Besides, my point was that one of the reasons why languages with (sound-ish) static types manage to have better performance because they can omit all of those run-time type checks (and the supporting machinery) because they'd never fail. And if you have to put those explicit checks, then the type hints are actually entirely redundant: e.g. Erlang's JIT ignores type specs, it instead looks at the type guards in the code to generate specialized code for the function bodies.
formerly_proven 3y ago
You can't really on type annotations to help interpret the code.
Difwif 3y ago
AFAIK good JITs like V8 can do runtime introspection and recompile on the fly if types change. Maybe using the type hints will be helpful but I don't think they are necessary for significant improvement.
- amelius 3y ago
  Are there any benchmarks that give an idea of how much this might improve Python's speed?
  - mike_hearn 3y ago
    Well, GraalPython is a Python JIT compiler which can exploit dynamically determined types, and it advertises 4.3x faster, so it's possible to do drastically better than a few percent. I think that's state of the art but might be wrong. That's for this benchmark: https://pyperformance.readthedocs.io/ https://pyperformance.readthedocs.io/ Note that this is with a relatively small investment as these things go, the GraalPython team is about ~3 people I guess, looking at the GH repo. It's an independent implementation so most of the work went into being compatible with Python including native extensions (the hard part). But this speedup depends a lot on what you're doing. Some types of code can go much faster. Others will be slower even than CPython, for example if you want to sandbox the native code extensions.
    - amelius 3y ago
      This is great info, thanks!
  - pletnes 3y ago
    Pypy is a different JIT that gives anything from slower/same to 100x speedup depending on the benchmark. They give a geometric mean of 4.8x speedup across their suite of benchmarks. https://speed.pypy.org/ https://speed.pypy.org/
- cuchoi 3y ago
  Doesn't Python already do this? https://www.youtube.com/watch?v=shQtrn1v7sQ https://www.youtube.com/watch?v=shQtrn1v7sQ
dataangel 3y ago
I doubt it with a copy-and-patch JIT, not the way they work now. I'm a serious mypy/python-static-types user and as is they currently wouldn't allow you to do much optimization wise. - All integers are still big integers - Use of the typing opt-out 'Any' is very common - All functions/methods can still be overwritten at runtime - Fields can still be added and removed from objects at runtime The combination basically makes it mandatory to not use native arithmetic, allocate everything on the heap, and need multiple levels of indirection for looking up any variable/field/function. CPU perf nightmare. You need a real optimizing JIT to track when integers are in a narrow range and things aren't getting redefined at runtime.
tekknolagi 3y ago
Sort of! But also not really. If you want to get into this, I wrote a post about this: https://bernsteinbear.com/blog/typed-python/ https://bernsteinbear.com/blog/typed-python/

Someone 3y ago

It should be fairly easy to add instruction fusing, where they recognize often-used instruction pairs, combine their C code, and then let the compiler optimize the combined code. Combining LOAD_CONST with the instruction following it if that instruction pops the const from the stack seems an easy win, for example.

ncruces 3y ago
If it was that easy, you'd do that in the interpreter and proportionally reduce interpretation overhead.
- Someone 3y ago
  In the interpreter, I don’t think it wouldn’t reduce overhead much, if at all. You’d still have to recognize the two byte codes, and your interpreter would spend additional time deciding, for most byte code pairs, that it doesn’t know how to combine them. With a compiler, that part is done once and, potentially, run zillions of times.
  - ncruces 3y ago
    If fusing a certain pair would significantly improve performance of most code, you'd just add that fused instruction to your bytecode and let the C compiler optimize the combined code in the interpreter. I have to assume CPython as already done that for all the low hanging fruit. In fact, for such a fused instruction to be optimized that way on a copy-and-patch JIT it'd need to exist as a new bytecode in interpreter. A JIT that fuses instructions is no longer a copy-and-patch JIT. A copy-and-patch JIT reduces interpretation overhead by making sure the branches in the executed machine code are the branches in the code to be interpreted, not branches in the interpreter. This is make a huge difference in more naive interpreters, not so much in an heavily optimized threaded-code interpreter. The 10% is great, and nothing to sneeze at for a first commit. But I'd actually like some realistic analysis of next steps for improvement, because I'm skeptical instruction fusing and other things being hand waved are it. Certainly not on a copy-and-patch JIT. For context: I spent significant effort trying to add such instruction fusing to a simple WASM AOT compiler and got nowhere (the equivalent of constant loading was precisely one of the pairs). Only moving to a much smarter JIT (capable of looking at whole basic blocks of instructions) started making a difference.

londons_explore 3y ago

> . I don't see an architectural path from a cut-and-paste JIT to something optimizing. One approach used in V8 is to have a dumb-but-very-fast JIT (ie. this), and keep counters of how often each block of code runs (perhaps actual counters, perhaps using CPU sampling features), and then any block of code running more than a few thousand times run through a far more complex yet slower optimizing jit. That has the benefit that the 0.2% of your code which uses 95% of the runtime is the only part that has to undergo the expensive optimization passes.

Sesse__ 3y ago
Note that V8 didn't have a dumb-but-very-fast JIT (Sparkplug) until 2021; the interpreter (Ignition) did that block counting and sent it straight to the optimizing JIT (TurboFan). V8 pre-2021 (i.e., only Ignition+TurboFan) was significantly faster than current CPython is, and the full current four-tier bundle (Ignition+Sparkplug+Maglev+TurboFan) only scores roughly twice as good on Speedometer as pure Ignition does. (Ignition+Sparkplug is about 40% faster than Ignition alone; compare that “dumbness” with CPython's 2–9%.) The relevant lesson should be that things like very carefully designed value representation and IR is a much more important piece of the puzzle than having as many tiers of compilation as possible.
- uluyol 3y ago
  In case anyone is interested, V8 pre-ignition/TurboFan had different tiers [1]: full-codegen (dumb and fast) and crankshaft (optimizing). It's interesting to see how these things change over time. [1]: https://v8.dev/blog/ignition-interpreter https://v8.dev/blog/ignition-interpreter
dataangel 3y ago
> keep counters of how often each block of code runs ... and then any block of code running more than a few thousand times run through a far more complex yet slower optimizing jit. That's just all JITs. Sometimes its counters for going from interpreter -> JIT rather than levels of JITs, but this idea is as old as JITs.