7 ms·

Performance of the Python 3.14 tail-call interpreter

607 points by signa11 2y ago

IshKebab 2y ago
Very interesting! Clearly something else going on though if the 2% Vs 9% thing is true.
jeeybee 2y ago
Kudos to the author for diving in and uncovering the real story here. The Python 3.14 tail-call interpreter is still a nice improvement (any few-percent gain in a language runtime is hard-won), just not a magic 15% free lunch. More importantly, this incident gave us valuable lessons about benchmarking rigor and the importance of testing across environments. It even helped surface a compiler bug that can now be fixed for everyone’s benefit. It’s the kind of deep-dive that makes you double-check the next big performance claim. Perhaps the most thought-provoking question is: how many other “X% faster” results out there are actually due to benchmarking artifacts or unknown regressions? And how can we better guard against these pitfalls in the future?
- ehsankia 2y ago
  I guess the bigger question for me is, how was a 10% drop in Python performance not detected when that faulty compiler feature was pushed? Do we not benchmark the compilers themselves? Do the existing benchmarks on the compiler or python side not use that specific compiler?
  - twoodfin 2y ago
    The author makes this point, too, and I agree it’s the most surprising thing about the entire scenario. LLVM introduced a major CPython performance regression, and nobody noticed for six months?
    - ltfish 2y ago
      As far as I am aware, the official CPython binaries on Linux have always been built using GCC, so you will have to build your own CPython using both Clang 18 and 19 to notice the speed difference. I think this is partly why no one has noticed the speed difference yet.
unit149 2y ago
[dead]
asicsp 2y ago
Related discussions: https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-tail-call https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-... --> https://news.ycombinator.com/item?id=42999672 https://news.ycombinator.com/item?id=42999672 (66 points | 25 days ago | 22 comments) https://blog.reverberate.org/2025/02/10/tail-call-updates.html https://blog.reverberate.org/2025/02/10/tail-call-updates.ht... --> https://news.ycombinator.com/item?id=43076088 https://news.ycombinator.com/item?id=43076088 (124 points | 18 days ago | 92 comments)
tempay 2y ago
Trying to assess the performance of a python build is extremely difficult as there are a lot of build tricks you can do to improve it. Recently the astral folks ran into this showing how the conda-forge build is notable faster than most others: https://github.com/astral-sh/python-build-standalone/pull/540 https://github.com/astral-sh/python-build-standalone/pull/54... I'd be interested to know how the tail-call interpreter performs with other build optimisations that exist.
- eru 2y ago
  Compare https://donsbot.com/2009/03/09/evolving-faster-haskell-programs/ https://donsbot.com/2009/03/09/evolving-faster-haskell-progr... The author uses a genetic algorithm to try out lots of different compiler and optimisation flag combinations.
vkazanov 2y ago
So, the compiler is tinkering with the way the loop is organised so the whole tail-call interpreter thing is not as effective as announced... Not surprised. 1. CPU arch (and arch version) matters a lot. The problem is 95% about laying out the instruction dispatching code for the branch predictor to work optimally. C was never meant to support this. 2. The C abstract machine is also not low-level enough to express the intent properly. Any implementation becomes supersensitivy to a particular compiler's (and compiler version) quirks. Certain paranoid interpreter implementation go back to writing assembly directly. luajit's famous for implementing a macro system to make its superefficient assembly loop implementation portable across architectures. This is also I find it fun to tinker with these! Anyways, a few years ago I've put together an article and a a test for popular interpreter loop implementation approaches: https://github.com/vkazanov/bytecode-interpreters-post https://github.com/vkazanov/bytecode-interpreters-post
- nelhage 2y ago
  (author here) > The problem is 95% about laying out the instruction dispatching code for the branch predictor to work optimally. A fun fact I learned while writing this post is that that's no longer true! Modern branch predictors can pretty much accurately predict through a single indirect jump, if the run is long enough and the interpreted code itself has stable behavior! Here's a paper that studied this (for both real hardware and a certain simulated branch predictor): https://inria.hal.science/hal-01100647/document https://inria.hal.science/hal-01100647/document My experiments on this project anecdotally agree; they didn't make it into the post but I also explored a few of the interpreters through hardware CPU counters and `perf stat`, and branch misprediction never showed up as a dominant factor.
  - vkazanov 2y ago
    Yes, this was already becoming true around the time I was writing the linked article. And I also read the paper. :-) I also remember I had access to a pre-Haswell era Intel CPUs vs something a bit more recent, and could see that the more complicated dispatcher no longer made as much sense. Conclusion: the rise of popular interpreter-based languages lead to CPUs with smarter branch predictors. What's interesting is that a token threaded interpreter dominated my benchmark (https://github.com/vkazanov/bytecode-interpreters-post/blob/984799b9c3337128f2b615826bc22efbd3d698e6/pigletvm.c#L424C1-L429C6 https://github.com/vkazanov/bytecode-interpreters-post/blob/...). This trick is meant to simplify dispatching logic and also spread branches in the code a bit.
kryptiskt 2y ago
This is a very good example of how C is not "close to the machine" or "portable assembly", modern optimizers will do drastic changes to the logic as long as it has no observable effect. As stated in the post: "Thus, we end up in this odd world where clang-19 compiles the computed-goto interpreter “correctly” – in the sense that the resulting binary produces all the same value we expect – but at the same time it produces an output completely at odds with the intention of the optimization. Moreover, we also see other versions of the compiler applying optimizations to the “naive” switch()-based interpreter, which implement the exact same optimization we “intended” to perform by rewriting the source code."
- fouronnes3 2y ago
  Stretching "no observable effect" all the way to a 10000 word blog post.
- jmillikin 2y ago
  > This is a very good example of how C is not "close to the machine" or > "portable assembly", C is very much "portable assembly" from the perspective of other systems programming languages of the 80s-90s era. The C expression `a += 1` can be trusted to increment a numeric value, but the same expression in C++ might allocate memory or unwind the call stack or do who knows what. Similarly, `a = "a"` is a simple pointer assignment in C, but in C++ it might allocate memory or [... etc]. The phrase "C is portable assembly" isn't a claim that each statement gets compiled directly to equivalent machine code.
  - eru 2y ago
    > The C expression `a += 1` can be trusted to increment a numeric value, [...] Have you heard of undefined behaviour?
    - jmillikin 2y ago
      Show me a C compiler that miscompiles the following code and I'll concede the point: uint32_t add_1(uint32_t a) { a += 1; return a; }
MattPalmer1086 2y ago
Benchmarking is just insanely hard to do well. There are so many things which can mislead you. I recently discovered a way to make an algorithm about 15% faster. At least, that's what all the benchmarks said. At some point I duplicated the faster function in my test harness, but did not call the faster version, just the original slower one... And it was still 15% faster. So code that never executed sped up the original code...!!! Obviously, this was due to code and memory layout issues, moving something so it aligned with some CPU cache better. It's actually really really hard to know if speedups you get are because your code is actually "better" or just because you lucked out with some better alignment somewhere. Casey Muratori has a really interesting series about things like this in his substack.
- eru 2y ago
  I vaguely remember about some benchmarking project that deliberately randomised these compiler decisions, so that they could give you more stable estimates of how well your code actually performed, and not just how well you won or lost the linker lottery.
  - Mond_ 2y ago
    You're probably thinking of "Performance Matters" by Emery Berger, a Strange Loops talk. https://youtube.com/watch?v=r-TLSBdHe1A https://youtube.com/watch?v=r-TLSBdHe1A
  - MattPalmer1086 2y ago
    There was Stabilizer [1] which did this, although it is no longer maintained and doesn't work with modern versions of LLVM. I think there is something more current now that automates this, but can't remember what it's called. [1] https://emeryberger.com/research/stabilizer/ https://emeryberger.com/research/stabilizer/
  - alpaca128 2y ago
    As already mentioned this is likely Emery Berger’s project with the idea of intentionally slowing down different parts of the program, also to find out which parts are most sensitive to slowdowns (aka have the biggest effect on overall performance), with the assumption that these are also the parts that profit the most from optimisations.
albertzeyer 2y ago
To clarify: The situation is still not completely understood? It's not just only the computed gotos, but there is some other regression in Clang19? Basically, the difference between clang19.nocg and clang19 is not really clear? Btw, what about some clang18.tc comparison, i.e. Clang18 with the new tail-call interpreter? I wonder how that compares to clang19.tc.
- aeyes 2y ago
  The article includes a link to the pull request which fixes the regression in LLVM: https://github.com/llvm/llvm-project/pull/114990 https://github.com/llvm/llvm-project/pull/114990
- nelhage 2y ago
  > Btw, what about some clang18.tc comparison (post author here) Oh, this is something I could have called out explicitly: The tail-calling interpreter relies on a feature (the `preserve_none` calling convention) that only landed in clang-19. That means you can only test it on that version. That coincidence (that 19 added both this feature, and the regression) is part of why this was so easy to miss at first, and why I had to "triangulate" with so many different benchmarks to be confident I understood what was going on.
thrdbndndn 2y ago
Great article! One detail caught my attention. In one of the referenced articles, https://simonwillison.net/2025/Feb/13/python-3140a5/ https://simonwillison.net/2025/Feb/13/python-3140a5/, the author wrote: "So 3.14.0a5 scored 1.12 times faster than 3.13 on the benchmark (on my extremely overloaded M2 MacBook Pro)." I'm quite confused by this. Did the author run the benchmark while the computer was overloaded with other processes? Wouldn't that make the results completely unreliable? I would have thought these benchmarks are conducted in highly controlled environments to eliminate external variables.
- ambivalence 2y ago
  Simon Willison is a great guy, but he's not a Python core developer and his ad hoc benchmark is not what the CPython core team members are using. For the latter, see https://github.com/faster-cpython/benchmarking-public https://github.com/faster-cpython/benchmarking-public
motbus3 2y ago
I recently made some benchmarking from python 3.9 to 3.13 And up to 3.11 it only got better. Python 3.12 and 3.13 were about 10% slower than 3.11. I thought my homemade benchmark wasn't great enough so I deployed it to a core service anyway and I saw same changes in our collected metrics. Does anyone else have the same problem?
- sgarland 2y ago
  Yes, I found a loop performance regression [0] in 3.12 and 3.13. [0]: https://github.com/python/cpython/issues/123540 https://github.com/python/cpython/issues/123540
- lazka 2y ago
  I'm also still using 3.11 for a FastAPI app since 3.12 and 3.13 are quite a bit slower.
cb321 2y ago
While some people here are citing 10% as "large" and 1% as "normal", there are optimizations like partial inlining of doubly recursive Fibonacci that can reduce the actual work (and so also time) exponentially (factors >10X for double-digit arguments or 1000s of %, technically exponential in the difference of the recursion depth, not the problem size [1]). C compilers can also be very finicky about their code inlining metrics. So, whether that enormous speed-up is realized can be very sensitive to the shape of your code. So, while part of the problem is definitely that CPUs have gotten quite fancy/complex, another aspect is that compilers "beyond -O0 or -O1" have also gotten fancy/complex. The article here, while good & worth reading, is also but one of many examples of how two complex things interacting can lead to very surprising results (and this is true outside of computing). People have a rather strong tendency to oversimplify -- no matter how many times this lesson shows up. EDIT: Also, the article at least uses two CPUs, Intel & Apple M1 and two compilers (gcc & clang), but there are realistic deployment scenarios across many more generations/impls of Intel, AMD, and ARM and probably other compilers, too. So, it only even samples a small part of the total complexity. Also, if you want to be more scientific, esp. for differences like "1.01X" then time measurements should really have error bars of some kind (either std.dev of the mean or maybe better for a case like this std.dev of the min[2]) and to minimize those measurement errors you probably want CPU core affinity scheduling in the OS. [1] https://stackoverflow.com/questions/360748/computational-complexity-of-fibonacci-sequence https://stackoverflow.com/questions/360748/computational-com... [2] https://github.com/c-blake/bu/blob/main/doc/tim.md https://github.com/c-blake/bu/blob/main/doc/tim.md
kenjin4096 2y ago
Hello. I'm the author of the PR that landed the tail-calling interpreter in CPython. First, I want to say thank you to Nelson for spending almost a month to get to the root of this issue. Secondly, I want to say I'm extremely embarrassed and sorry that I made such a huge oversight. I, and probably the rest of the CPython team did not expect the compiler we were using for the baseline to have that bug. I posted an apology blog post here. https://fidget-spinner.github.io/posts/apology-tail-call.html https://fidget-spinner.github.io/posts/apology-tail-call.htm...
- jxjnskkzxxhx 2y ago
  I just wanted to say: respect for being able to say "sorry, I made a mistake". I hate the fake it till you make it mentality that seems to be the norm now.
  - kzrdude 2y ago
    I understand the frustration but I don't think it needed to be said (the part about mentality, the thanks is of course cool), because that's still not the norm. Why do I even bring this message - I want to say that let's not let what we see in the news influence our perception of the real people of the world. Just because fraud and crimes get elevated in the news, does not mean that the common man is a criminal or a fraud. :)
    - codr7 2y ago
      Divide and conquer; we're supposed to hate each other and trust the state/elite/technology, at least that's the plan. The real criminals, the people we should keep an eye on, are the plan's designers and implementers.
- ptx 2y ago
  Why didn't this regression in baseline performance show up (or did it?) on the faster-cpython benchmarks page [0]? Could the benchmarks be improved to prevent similar issues in the future? [0] https://github.com/faster-cpython/benchmarking-public https://github.com/faster-cpython/benchmarking-public
bjourne 2y ago
This is exactly the kind of content I love to see on HN. But I wonder though how this optimization is related to tail-call optimization? How the interpreter jump table is implemented shouldn't affect how stack frames are created, should it?
- mbel 2y ago
  It's explained under the first link from the article [0]: "A new type of interpreter has been added to CPython. It uses tail calls between small C functions that implement individual Python opcodes, rather than one large C case statement." [0] https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-tail-call https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-...
- ambivalence 2y ago
  A big chunk of the performance gain comes from the fact that with tail calls the CPU doesn't have to reset the registers (configured through the `preserve_none` calling convention).
- thunkingdeep 2y ago
  Well if you’d bother to read it, you’d discover this is about tail calls in C, not in Python. It has nothing to do with tail recursion in Python. Guido has explicitly said that Python will never have it.
deleted 2y ago
[deleted]
throw132128 2y ago
That is what happens in a project if you purge all opposition and a few people work on corporate projects that are supposed to succeed. The free-threading had massive regressions (50% slower), this one has almost no speedups. People are scared of corporate repressions or repressions from the Steering Council, so they shut up like in the Soviet Union.
Retr0id 2y ago
It's getting harder to characterize the C language as "close to the metal". Obviously optimizing compilers are a good thing, but it's frustrating when it feels like you're fighting with them to get the code emitted just the way you want.
Malp 2y ago
This is some solid work and a great blog post to go hand-in-hand with. Wow.
AlexSW 2y ago
I'm struggling to confirm that my intuition of why the compiler optimisation is faster than the naive switch expansion. Is it about the layout of where the assembly instructions end up, and spacing around them? Or the CPU pipelining working better? Or...?
- csjh 2y ago
  distributing the dispatch instructions lets the predictor pick up on specific patterns within the bytecode - for example, comparison instructions are probably followed by a conditional branch instruction
  - AlexSW 2y ago
    Ah of course! It's improving the branch target predictor. Makes sense, thanks!
anarazel 2y ago
Hah, I found something similar in gcc a few years back. Affecting postgres' expression evaluation interpreter: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71785 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71785
ram_rar 2y ago
Kudo to author and Cpython team to reckon this. Nevertheless, its still a very important improvement in +Ve direction. Thats all matters. This a great story as to why benchmarking are soo hard to get right. I always tell my team to benchmark your real world use case as closely as possible and not rely on external results.
ashvardanian 2y ago
Thanks for the post! Would love to read similarly styled dives into GIL performance and experimental multi-threaded Python builds!
ta20211004_1 2y ago
Congratulations! You have nothing to apologize for.
cytocync 2y ago
[dead]
ltbarcly3 2y ago
Why hasn't anyone just carbon copied the Python compiler (coincidentally the same name but unrelated to the Python language) from SBCL? The semantics of CL aren't that different from Python (the language).
- gjvc 2y ago
  The compiler is named Python, but has nothing to do with Python the language. I mean __really__. Way to keep your invention buried.
  - ltbarcly3 2y ago
    Because they started work on Python (the compiler) in 1980!
ltbarcly3 2y ago
I think it's remarkable that nobody thought to benchmark against a independent build, maybe from debian or redhat. I do this for my local builds I use for development even, it's just so absurdly obvious (and not just in hindsight). No real harm was done but some embarrassment, but it's pretty silly.
Cheng2023 2y ago
[dead]

607 points by signa11 2y ago

Very interesting! Clearly something else going on though if the 2% Vs 9% thing is true.

Kudos to the author for diving in and uncovering the real story here. The Python 3.14 tail-call interpreter is still a nice improvement (any few-percent gain in a language runtime is hard-won), just not a magic 15% free lunch. More importantly, this incident gave us valuable lessons about benchmarking rigor and the importance of testing across environments. It even helped surface a compiler bug that can now be fixed for everyone’s benefit. It’s the kind of deep-dive that makes you double-check the next big performance claim. Perhaps the most thought-provoking question is: how many other “X% faster” results out there are actually due to benchmarking artifacts or unknown regressions? And how can we better guard against these pitfalls in the future?

ehsankia 2y ago
I guess the bigger question for me is, how was a 10% drop in Python performance not detected when that faulty compiler feature was pushed? Do we not benchmark the compilers themselves? Do the existing benchmarks on the compiler or python side not use that specific compiler?
- twoodfin 2y ago
  The author makes this point, too, and I agree it’s the most surprising thing about the entire scenario. LLVM introduced a major CPython performance regression, and nobody noticed for six months?
  - ltfish 2y ago
    As far as I am aware, the official CPython binaries on Linux have always been built using GCC, so you will have to build your own CPython using both Clang 18 and 19 to notice the speed difference. I think this is partly why no one has noticed the speed difference yet.

unit149 2y ago

[dead]

asicsp 2y ago

Related discussions: https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-tail-call https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-... --> https://news.ycombinator.com/item?id=42999672 https://news.ycombinator.com/item?id=42999672 (66 points | 25 days ago | 22 comments) https://blog.reverberate.org/2025/02/10/tail-call-updates.html https://blog.reverberate.org/2025/02/10/tail-call-updates.ht... --> https://news.ycombinator.com/item?id=43076088 https://news.ycombinator.com/item?id=43076088 (124 points | 18 days ago | 92 comments)

tempay 2y ago

Trying to assess the performance of a python build is extremely difficult as there are a lot of build tricks you can do to improve it. Recently the astral folks ran into this showing how the conda-forge build is notable faster than most others: https://github.com/astral-sh/python-build-standalone/pull/540 https://github.com/astral-sh/python-build-standalone/pull/54... I'd be interested to know how the tail-call interpreter performs with other build optimisations that exist.

eru 2y ago
Compare https://donsbot.com/2009/03/09/evolving-faster-haskell-programs/ https://donsbot.com/2009/03/09/evolving-faster-haskell-progr... The author uses a genetic algorithm to try out lots of different compiler and optimisation flag combinations.

vkazanov 2y ago

So, the compiler is tinkering with the way the loop is organised so the whole tail-call interpreter thing is not as effective as announced... Not surprised. 1. CPU arch (and arch version) matters a lot. The problem is 95% about laying out the instruction dispatching code for the branch predictor to work optimally. C was never meant to support this. 2. The C abstract machine is also not low-level enough to express the intent properly. Any implementation becomes supersensitivy to a particular compiler's (and compiler version) quirks. Certain paranoid interpreter implementation go back to writing assembly directly. luajit's famous for implementing a macro system to make its superefficient assembly loop implementation portable across architectures. This is also I find it fun to tinker with these! Anyways, a few years ago I've put together an article and a a test for popular interpreter loop implementation approaches: https://github.com/vkazanov/bytecode-interpreters-post https://github.com/vkazanov/bytecode-interpreters-post

nelhage 2y ago
(author here) > The problem is 95% about laying out the instruction dispatching code for the branch predictor to work optimally. A fun fact I learned while writing this post is that that's no longer true! Modern branch predictors can pretty much accurately predict through a single indirect jump, if the run is long enough and the interpreted code itself has stable behavior! Here's a paper that studied this (for both real hardware and a certain simulated branch predictor): https://inria.hal.science/hal-01100647/document https://inria.hal.science/hal-01100647/document My experiments on this project anecdotally agree; they didn't make it into the post but I also explored a few of the interpreters through hardware CPU counters and `perf stat`, and branch misprediction never showed up as a dominant factor.
- vkazanov 2y ago
  Yes, this was already becoming true around the time I was writing the linked article. And I also read the paper. :-) I also remember I had access to a pre-Haswell era Intel CPUs vs something a bit more recent, and could see that the more complicated dispatcher no longer made as much sense. Conclusion: the rise of popular interpreter-based languages lead to CPUs with smarter branch predictors. What's interesting is that a token threaded interpreter dominated my benchmark (https://github.com/vkazanov/bytecode-interpreters-post/blob/984799b9c3337128f2b615826bc22efbd3d698e6/pigletvm.c#L424C1-L429C6 https://github.com/vkazanov/bytecode-interpreters-post/blob/...). This trick is meant to simplify dispatching logic and also spread branches in the code a bit.

kryptiskt 2y ago

This is a very good example of how C is not "close to the machine" or "portable assembly", modern optimizers will do drastic changes to the logic as long as it has no observable effect. As stated in the post: "Thus, we end up in this odd world where clang-19 compiles the computed-goto interpreter “correctly” – in the sense that the resulting binary produces all the same value we expect – but at the same time it produces an output completely at odds with the intention of the optimization. Moreover, we also see other versions of the compiler applying optimizations to the “naive” switch()-based interpreter, which implement the exact same optimization we “intended” to perform by rewriting the source code."

fouronnes3 2y ago
Stretching "no observable effect" all the way to a 10000 word blog post.
jmillikin 2y ago
> This is a very good example of how C is not "close to the machine" or > "portable assembly", C is very much "portable assembly" from the perspective of other systems programming languages of the 80s-90s era. The C expression `a += 1` can be trusted to increment a numeric value, but the same expression in C++ might allocate memory or unwind the call stack or do who knows what. Similarly, `a = "a"` is a simple pointer assignment in C, but in C++ it might allocate memory or [... etc]. The phrase "C is portable assembly" isn't a claim that each statement gets compiled directly to equivalent machine code.
- eru 2y ago
  > The C expression `a += 1` can be trusted to increment a numeric value, [...] Have you heard of undefined behaviour?
  - jmillikin 2y ago
    Show me a C compiler that miscompiles the following code and I'll concede the point: uint32_t add_1(uint32_t a) { a += 1; return a; }

MattPalmer1086 2y ago

Benchmarking is just insanely hard to do well. There are so many things which can mislead you. I recently discovered a way to make an algorithm about 15% faster. At least, that's what all the benchmarks said. At some point I duplicated the faster function in my test harness, but did not call the faster version, just the original slower one... And it was still 15% faster. So code that never executed sped up the original code...!!! Obviously, this was due to code and memory layout issues, moving something so it aligned with some CPU cache better. It's actually really really hard to know if speedups you get are because your code is actually "better" or just because you lucked out with some better alignment somewhere. Casey Muratori has a really interesting series about things like this in his substack.

eru 2y ago
I vaguely remember about some benchmarking project that deliberately randomised these compiler decisions, so that they could give you more stable estimates of how well your code actually performed, and not just how well you won or lost the linker lottery.
- Mond_ 2y ago
  You're probably thinking of "Performance Matters" by Emery Berger, a Strange Loops talk. https://youtube.com/watch?v=r-TLSBdHe1A https://youtube.com/watch?v=r-TLSBdHe1A
- MattPalmer1086 2y ago
  There was Stabilizer [1] which did this, although it is no longer maintained and doesn't work with modern versions of LLVM. I think there is something more current now that automates this, but can't remember what it's called. [1] https://emeryberger.com/research/stabilizer/ https://emeryberger.com/research/stabilizer/
- alpaca128 2y ago
  As already mentioned this is likely Emery Berger’s project with the idea of intentionally slowing down different parts of the program, also to find out which parts are most sensitive to slowdowns (aka have the biggest effect on overall performance), with the assumption that these are also the parts that profit the most from optimisations.

albertzeyer 2y ago

To clarify: The situation is still not completely understood? It's not just only the computed gotos, but there is some other regression in Clang19? Basically, the difference between clang19.nocg and clang19 is not really clear? Btw, what about some clang18.tc comparison, i.e. Clang18 with the new tail-call interpreter? I wonder how that compares to clang19.tc.

aeyes 2y ago
The article includes a link to the pull request which fixes the regression in LLVM: https://github.com/llvm/llvm-project/pull/114990 https://github.com/llvm/llvm-project/pull/114990
nelhage 2y ago
> Btw, what about some clang18.tc comparison (post author here) Oh, this is something I could have called out explicitly: The tail-calling interpreter relies on a feature (the `preserve_none` calling convention) that only landed in clang-19. That means you can only test it on that version. That coincidence (that 19 added both this feature, and the regression) is part of why this was so easy to miss at first, and why I had to "triangulate" with so many different benchmarks to be confident I understood what was going on.

thrdbndndn 2y ago

Great article! One detail caught my attention. In one of the referenced articles, https://simonwillison.net/2025/Feb/13/python-3140a5/ https://simonwillison.net/2025/Feb/13/python-3140a5/, the author wrote: "So 3.14.0a5 scored 1.12 times faster than 3.13 on the benchmark (on my extremely overloaded M2 MacBook Pro)." I'm quite confused by this. Did the author run the benchmark while the computer was overloaded with other processes? Wouldn't that make the results completely unreliable? I would have thought these benchmarks are conducted in highly controlled environments to eliminate external variables.

ambivalence 2y ago
Simon Willison is a great guy, but he's not a Python core developer and his ad hoc benchmark is not what the CPython core team members are using. For the latter, see https://github.com/faster-cpython/benchmarking-public https://github.com/faster-cpython/benchmarking-public

motbus3 2y ago

I recently made some benchmarking from python 3.9 to 3.13 And up to 3.11 it only got better. Python 3.12 and 3.13 were about 10% slower than 3.11. I thought my homemade benchmark wasn't great enough so I deployed it to a core service anyway and I saw same changes in our collected metrics. Does anyone else have the same problem?

sgarland 2y ago
Yes, I found a loop performance regression [0] in 3.12 and 3.13. [0]: https://github.com/python/cpython/issues/123540 https://github.com/python/cpython/issues/123540
lazka 2y ago
I'm also still using 3.11 for a FastAPI app since 3.12 and 3.13 are quite a bit slower.

cb321 2y ago

While some people here are citing 10% as "large" and 1% as "normal", there are optimizations like partial inlining of doubly recursive Fibonacci that can reduce the actual work (and so also time) exponentially (factors >10X for double-digit arguments or 1000s of %, technically exponential in the difference of the recursion depth, not the problem size [1]). C compilers can also be very finicky about their code inlining metrics. So, whether that enormous speed-up is realized can be very sensitive to the shape of your code. So, while part of the problem is definitely that CPUs have gotten quite fancy/complex, another aspect is that compilers "beyond -O0 or -O1" have also gotten fancy/complex. The article here, while good & worth reading, is also but one of many examples of how two complex things interacting can lead to very surprising results (and this is true outside of computing). People have a rather strong tendency to oversimplify -- no matter how many times this lesson shows up. EDIT: Also, the article at least uses two CPUs, Intel & Apple M1 and two compilers (gcc & clang), but there are realistic deployment scenarios across many more generations/impls of Intel, AMD, and ARM and probably other compilers, too. So, it only even samples a small part of the total complexity. Also, if you want to be more scientific, esp. for differences like "1.01X" then time measurements should really have error bars of some kind (either std.dev of the mean or maybe better for a case like this std.dev of the min[2]) and to minimize those measurement errors you probably want CPU core affinity scheduling in the OS. [1] https://stackoverflow.com/questions/360748/computational-complexity-of-fibonacci-sequence https://stackoverflow.com/questions/360748/computational-com... [2] https://github.com/c-blake/bu/blob/main/doc/tim.md https://github.com/c-blake/bu/blob/main/doc/tim.md

kenjin4096 2y ago

Hello. I'm the author of the PR that landed the tail-calling interpreter in CPython. First, I want to say thank you to Nelson for spending almost a month to get to the root of this issue. Secondly, I want to say I'm extremely embarrassed and sorry that I made such a huge oversight. I, and probably the rest of the CPython team did not expect the compiler we were using for the baseline to have that bug. I posted an apology blog post here. https://fidget-spinner.github.io/posts/apology-tail-call.html https://fidget-spinner.github.io/posts/apology-tail-call.htm...

jxjnskkzxxhx 2y ago
I just wanted to say: respect for being able to say "sorry, I made a mistake". I hate the fake it till you make it mentality that seems to be the norm now.
- kzrdude 2y ago
  I understand the frustration but I don't think it needed to be said (the part about mentality, the thanks is of course cool), because that's still not the norm. Why do I even bring this message - I want to say that let's not let what we see in the news influence our perception of the real people of the world. Just because fraud and crimes get elevated in the news, does not mean that the common man is a criminal or a fraud. :)
  - codr7 2y ago
    Divide and conquer; we're supposed to hate each other and trust the state/elite/technology, at least that's the plan. The real criminals, the people we should keep an eye on, are the plan's designers and implementers.
ptx 2y ago
Why didn't this regression in baseline performance show up (or did it?) on the faster-cpython benchmarks page [0]? Could the benchmarks be improved to prevent similar issues in the future? [0] https://github.com/faster-cpython/benchmarking-public https://github.com/faster-cpython/benchmarking-public

bjourne 2y ago

This is exactly the kind of content I love to see on HN. But I wonder though how this optimization is related to tail-call optimization? How the interpreter jump table is implemented shouldn't affect how stack frames are created, should it?

mbel 2y ago
It's explained under the first link from the article [0]: "A new type of interpreter has been added to CPython. It uses tail calls between small C functions that implement individual Python opcodes, rather than one large C case statement." [0] https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-tail-call https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-...
ambivalence 2y ago
A big chunk of the performance gain comes from the fact that with tail calls the CPU doesn't have to reset the registers (configured through the `preserve_none` calling convention).
thunkingdeep 2y ago
Well if you’d bother to read it, you’d discover this is about tail calls in C, not in Python. It has nothing to do with tail recursion in Python. Guido has explicitly said that Python will never have it.

deleted 2y ago

[deleted]

throw132128 2y ago

That is what happens in a project if you purge all opposition and a few people work on corporate projects that are supposed to succeed. The free-threading had massive regressions (50% slower), this one has almost no speedups. People are scared of corporate repressions or repressions from the Steering Council, so they shut up like in the Soviet Union.

Retr0id 2y ago

It's getting harder to characterize the C language as "close to the metal". Obviously optimizing compilers are a good thing, but it's frustrating when it feels like you're fighting with them to get the code emitted just the way you want.

Malp 2y ago

This is some solid work and a great blog post to go hand-in-hand with. Wow.

AlexSW 2y ago

I'm struggling to confirm that my intuition of why the compiler optimisation is faster than the naive switch expansion. Is it about the layout of where the assembly instructions end up, and spacing around them? Or the CPU pipelining working better? Or...?

csjh 2y ago
distributing the dispatch instructions lets the predictor pick up on specific patterns within the bytecode - for example, comparison instructions are probably followed by a conditional branch instruction
- AlexSW 2y ago
  Ah of course! It's improving the branch target predictor. Makes sense, thanks!

anarazel 2y ago

Hah, I found something similar in gcc a few years back. Affecting postgres' expression evaluation interpreter: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71785 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71785

ram_rar 2y ago

Kudo to author and Cpython team to reckon this. Nevertheless, its still a very important improvement in +Ve direction. Thats all matters. This a great story as to why benchmarking are soo hard to get right. I always tell my team to benchmark your real world use case as closely as possible and not rely on external results.

ashvardanian 2y ago

Thanks for the post! Would love to read similarly styled dives into GIL performance and experimental multi-threaded Python builds!

ta20211004_1 2y ago

Congratulations! You have nothing to apologize for.

cytocync 2y ago

[dead]

ltbarcly3 2y ago

Why hasn't anyone just carbon copied the Python compiler (coincidentally the same name but unrelated to the Python language) from SBCL? The semantics of CL aren't that different from Python (the language).

gjvc 2y ago
The compiler is named Python, but has nothing to do with Python the language. I mean __really__. Way to keep your invention buried.
- ltbarcly3 2y ago
  Because they started work on Python (the compiler) in 1980!

ltbarcly3 2y ago

I think it's remarkable that nobody thought to benchmark against a independent build, maybe from debian or redhat. I do this for my local builds I use for development even, it's just so absurdly obvious (and not just in hindsight). No real harm was done but some embarrassment, but it's pretty silly.

Cheng2023 2y ago

[dead]