5 ms·

The article is a fine example of incremental optimization of some Python, replacing constructs that the standard Python interpreter has overheads in executing w

The article is a fine example of incremental optimization of some Python, replacing constructs that the standard Python interpreter has overheads in executing with others which trigger fewer of those same overheads. The title isn't quite right, though. Boxing, method lookup, etc. come under "interpreter" too. There's a continuum of implementation between a naive interpreter and a full-blown JIT compiler, rather than a binary distinction. All interpreters beyond line-oriented things like shells convert source code into a different format more suitable for execution. When that format is in memory, it can be processed to different degrees to achieve different levels of performance. For example, if looking up "str" is an issue, an interpreter could cache the last function it got for "str" conditional on some invalidation token (e.g. "global_identifier_change_count"), so it doesn't in fact need to look it up every time. Boxing can be eliminated by using type-specific representations of the operations, and choosing different code paths depending on checking the actual type. Hoisting those type checks outside of loops is then a big win. Add inlining, and the hoisting logic can see more loops to move things out of. Inlining also gets rid of your argument passing overhead. None of this requires targeting machine code directly, you can optimize interpreter code, and in fact that's what the back end of retargetable optimizers looks like - intermediate representation is an interpretable format. Of course things get super-complex super-quickly, but that's the trade-off.

chrisseaton 6y ago
> There's a continuum implementations between a naive interpreter and a full-blown JIT compiler And many full-blown JIT compilers also include an interpreter for things like deoptimisation and rolling towards a safe state. Few things are a pure JIT (I think the .NET runtime is) and actually you don't want that is limits how far you can optimise.
- gsnedders 6y ago
  > Few things are a pure JIT V8 was notably a pure JIT for most of its existence, though did relatively recently get an interpreter. (Previously it had a baseline template JIT.)
- whizzter 6y ago
  It always depends on the use-case and language characteristics. V8 used to be JIT-only for a long time (generating naive x86 code isn't much slower than bytecode), IIRC a big the reason they added a interpreter mode was partly to decrease memory pressure since JS usage had gone through the roof over the years (and the fact that most of the JS code was ever only run one time). Also faster JIT compilers for JS,etc makes many special cases and paths for type-specialization to avoid GC pressure,etc whereas a .NET or Java runtime has mostly primtive-static paths(Java pre-invokedynamic), ie most of the time an int, double or object reference can't appear in the same "variable slot". The particular object type matters more for invocation optimizations(especially when it comes to interfaces) but less for data movement in generic code.
  - gsnedders 6y ago
    > IIRC a big the reason they added a interpreter mode was partly to decrease memory pressure since JS usage had gone through the roof over the years There were a whole bunch of reasons: IIRC, average memory on devices Chrome runs on was _decreasing_ over time (due to growth in low-end mobile devices) while average JS code size was increasing. Memory consumption from the relatively naïvely generated machine code was sizeable. The baseline JIT was based on the original V8 JIT, which was becoming a maintenance nightmare (it was no longer used for any optimised tier, IIRC), in part because it was very much originally written for a single architecture and then ported elsewhere, and the per-architecture code-size was massive compared with the newer JITs. Increasing technical debt in the original JIT. Reducing the amount of duplicate implementation of new language features across various tiers.
acqq 6y ago
> The title isn't quite right, though. Boxing, method lookup, etc. come under "interpreter" too. See also jsnell's answer here, however. E.g. (with my interpretation added, all possible errors in them mine) a "naive compiler would have exactly the same boxing overhead" (during the execution)." And on the other hand a smart interpreter could eliminate it" (if it is programmed to recognize these cases during the execution). To add to that, if the previous processing of the source allows that during the interpretation phase some work can be done out of the loop, then it's not technically that "interpretation" that is sped up but the speedup is achieved by what having the "interpreter" code doing less work. Arguing from your perspective, one can call "an interpreter" everything that has an input of the source and outputs the result. But I personally, having worked on the compilers, believe that even the internals of "interpreted" languages of today have to be observed with more granularity. As an example, in the broader Perl world it's fully normal to speak about the "compiler" of Perl. What "comes under interpreter" depends on whom you ask. I believe that Python can be made faster than it is now, without other disadvantages but having maybe the sources of Python in some "less easy to understand" form, and I believe the post we all comment illustrates that. "Common" operations could have less default overhead, however we call that all.
kmod 6y ago
I have to disagree that those are "interpreter" overheads, since the later versions of the benchmark do not use the Python interpreter at all yet still suffer from these overheads. Maybe disagreeing on the wording is pedantic though, since I think the real discussion is what can be done about it. We definitely agree directionally: you can create specific implementations of operations that are fast for their inputs. But look at how specific we have to get here. It's not just on the type of the callable: we have to know the exact callable we are calling (the str type). And its behavior is heavily dependent on its argument type. So we need a code path that is specifically for calling the str() function on ints. I would argue that this is prohibitively specialized for an interpreter, but one can imagine a JIT that is able to produce this kind of specialization, and that's exactly what this blog post is trying to motivate.
- acqq 6y ago
  > So we need a code path that is specifically for calling the str() function on ints. I would argue that this is prohibitively specialized for an interpreter, I would argue it isn't. It's actually less about the ints but about the tuple allocation: "Now that we've removed enough code we can finally make another big optimization: no longer allocating the args tuple This cuts runtime down to 1.15s." (down from 1.40s) It seems to me that having special case for one argument, avoiding the tuple allocation in each call is nothing "prohibitively expensive" and would benefit all the functions being called with one argument, and there are enough of such. Regarding ints vs something else -- somewhere in the code there is anyway different code that does str of int vs str of something else. It's just about accessing that code for that type, it doesn't have to cost having today's speculative execution in the CPU's. The people who produce compiled code know about that and can use it.
  - kmod 6y ago
    There's a minor but important point: the amount of work str() has to do depends on its input type. In particular, if arg.__str__() returns a string or a subtype of string or not a string, str() will call different tp_init functions. So we can't get rid of the args tuple until after the optimization of knowing that arg.__str__() returns an string (which is why I did them in that order). So I guess the condition is a little bit weaker than str(int) but it's more than str() of one arg.
mamcx 6y ago
> you can optimize interpreter code I love to have a source where this optimizations are explained/show. I'm building mine, and the only way you found it is by luck looking at some codebases.
moonchild 6y ago
Every time someone talks about inherent interpreter overhead, I like to remind of k[1][2], which is faster than c despite being a relatively naive interpreter. 1. https://kx.com/ https://kx.com/ 2. https://shakti.com/ https://shakti.com/
- tom_mellior 6y ago
  Python is also blazingly fast when your Python program spends 99% of its time outside of Python and inside some highly optimized library like NumPy. That's because the interpreter overhead doesn't matter when you are not interpreting.
- clarry 6y ago
  It doesn't make much sense to say that an application written in C is faster than C (unless it includes a compiler and recompiles itself..). Ok, well, they probably aren't using C today, but I doubt their implementation language is significantly faster than C.
amrx101 6y ago
As someone who has wrote Python professionally here is my quirk about it. As soon as performance requirements start cropping up in any Python project, all "easy-ness" of Python starts to fade. You have to start looking up for optimisations left, right and centre. Re-writing your code with C-calls to improve performance. All these has taught me a thing or two. Now if I know that we will be using the product wherein performance or scalability is required, I straight away opt one of Rust, Golang or JAVA. I dont want recurring efforts on tuning the code-base to get better performance. I know I might get some opinions here, but Python is a scripting language for rapid prototyping and I plan to use it as such.
- sumanthvepa 6y ago
  Yup. Use the right tool for the job. Python's performance is definitely something that the implementers of Python should worry about, but as a user, you have choices and you should exercise them. If your problem requires relatively high performance: Java, Go and Rust are all good alternatives. If absolute pedal-to-the-metal, no compromises, performance is needed: C, C++ are your choices. Heck, you can even inline assembly in some cases.
- Areading314 6y ago
  This is especially true when writing code that needs to run on many cores -- Python starts getting really cumbersome in this case, which can be avoided using the relatively elegant threading models in alternative languages.
  - jashmatthews 6y ago
    Task based parallelism for a web server or background job server is really easy and that's 99% of the cases covered. Problems which _need_ full multi-threading with shared mutable state are actually pretty rare.
    - Areading314 6y ago
      Agreed -- but they do happen!

chrisseaton 6y ago

> There's a continuum implementations between a naive interpreter and a full-blown JIT compiler And many full-blown JIT compilers also include an interpreter for things like deoptimisation and rolling towards a safe state. Few things are a pure JIT (I think the .NET runtime is) and actually you don't want that is limits how far you can optimise.

gsnedders 6y ago
> Few things are a pure JIT V8 was notably a pure JIT for most of its existence, though did relatively recently get an interpreter. (Previously it had a baseline template JIT.)
whizzter 6y ago
It always depends on the use-case and language characteristics. V8 used to be JIT-only for a long time (generating naive x86 code isn't much slower than bytecode), IIRC a big the reason they added a interpreter mode was partly to decrease memory pressure since JS usage had gone through the roof over the years (and the fact that most of the JS code was ever only run one time). Also faster JIT compilers for JS,etc makes many special cases and paths for type-specialization to avoid GC pressure,etc whereas a .NET or Java runtime has mostly primtive-static paths(Java pre-invokedynamic), ie most of the time an int, double or object reference can't appear in the same "variable slot". The particular object type matters more for invocation optimizations(especially when it comes to interfaces) but less for data movement in generic code.
- gsnedders 6y ago
  > IIRC a big the reason they added a interpreter mode was partly to decrease memory pressure since JS usage had gone through the roof over the years There were a whole bunch of reasons: IIRC, average memory on devices Chrome runs on was _decreasing_ over time (due to growth in low-end mobile devices) while average JS code size was increasing. Memory consumption from the relatively naïvely generated machine code was sizeable. The baseline JIT was based on the original V8 JIT, which was becoming a maintenance nightmare (it was no longer used for any optimised tier, IIRC), in part because it was very much originally written for a single architecture and then ported elsewhere, and the per-architecture code-size was massive compared with the newer JITs. Increasing technical debt in the original JIT. Reducing the amount of duplicate implementation of new language features across various tiers.

acqq 6y ago

> The title isn't quite right, though. Boxing, method lookup, etc. come under "interpreter" too. See also jsnell's answer here, however. E.g. (with my interpretation added, all possible errors in them mine) a "naive compiler would have exactly the same boxing overhead" (during the execution)." And on the other hand a smart interpreter could eliminate it" (if it is programmed to recognize these cases during the execution). To add to that, if the previous processing of the source allows that during the interpretation phase some work can be done out of the loop, then it's not technically that "interpretation" that is sped up but the speedup is achieved by what having the "interpreter" code doing less work. Arguing from your perspective, one can call "an interpreter" everything that has an input of the source and outputs the result. But I personally, having worked on the compilers, believe that even the internals of "interpreted" languages of today have to be observed with more granularity. As an example, in the broader Perl world it's fully normal to speak about the "compiler" of Perl. What "comes under interpreter" depends on whom you ask. I believe that Python can be made faster than it is now, without other disadvantages but having maybe the sources of Python in some "less easy to understand" form, and I believe the post we all comment illustrates that. "Common" operations could have less default overhead, however we call that all.

kmod 6y ago

I have to disagree that those are "interpreter" overheads, since the later versions of the benchmark do not use the Python interpreter at all yet still suffer from these overheads. Maybe disagreeing on the wording is pedantic though, since I think the real discussion is what can be done about it. We definitely agree directionally: you can create specific implementations of operations that are fast for their inputs. But look at how specific we have to get here. It's not just on the type of the callable: we have to know the exact callable we are calling (the str type). And its behavior is heavily dependent on its argument type. So we need a code path that is specifically for calling the str() function on ints. I would argue that this is prohibitively specialized for an interpreter, but one can imagine a JIT that is able to produce this kind of specialization, and that's exactly what this blog post is trying to motivate.

acqq 6y ago
> So we need a code path that is specifically for calling the str() function on ints. I would argue that this is prohibitively specialized for an interpreter, I would argue it isn't. It's actually less about the ints but about the tuple allocation: "Now that we've removed enough code we can finally make another big optimization: no longer allocating the args tuple This cuts runtime down to 1.15s." (down from 1.40s) It seems to me that having special case for one argument, avoiding the tuple allocation in each call is nothing "prohibitively expensive" and would benefit all the functions being called with one argument, and there are enough of such. Regarding ints vs something else -- somewhere in the code there is anyway different code that does str of int vs str of something else. It's just about accessing that code for that type, it doesn't have to cost having today's speculative execution in the CPU's. The people who produce compiled code know about that and can use it.
- kmod 6y ago
  There's a minor but important point: the amount of work str() has to do depends on its input type. In particular, if arg.__str__() returns a string or a subtype of string or not a string, str() will call different tp_init functions. So we can't get rid of the args tuple until after the optimization of knowing that arg.__str__() returns an string (which is why I did them in that order). So I guess the condition is a little bit weaker than str(int) but it's more than str() of one arg.

mamcx 6y ago

> you can optimize interpreter code I love to have a source where this optimizations are explained/show. I'm building mine, and the only way you found it is by luck looking at some codebases.

moonchild 6y ago

Every time someone talks about inherent interpreter overhead, I like to remind of k[1][2], which is faster than c despite being a relatively naive interpreter. 1. https://kx.com/ https://kx.com/ 2. https://shakti.com/ https://shakti.com/

tom_mellior 6y ago
Python is also blazingly fast when your Python program spends 99% of its time outside of Python and inside some highly optimized library like NumPy. That's because the interpreter overhead doesn't matter when you are not interpreting.
clarry 6y ago
It doesn't make much sense to say that an application written in C is faster than C (unless it includes a compiler and recompiles itself..). Ok, well, they probably aren't using C today, but I doubt their implementation language is significantly faster than C.

amrx101 6y ago

As someone who has wrote Python professionally here is my quirk about it. As soon as performance requirements start cropping up in any Python project, all "easy-ness" of Python starts to fade. You have to start looking up for optimisations left, right and centre. Re-writing your code with C-calls to improve performance. All these has taught me a thing or two. Now if I know that we will be using the product wherein performance or scalability is required, I straight away opt one of Rust, Golang or JAVA. I dont want recurring efforts on tuning the code-base to get better performance. I know I might get some opinions here, but Python is a scripting language for rapid prototyping and I plan to use it as such.

sumanthvepa 6y ago
Yup. Use the right tool for the job. Python's performance is definitely something that the implementers of Python should worry about, but as a user, you have choices and you should exercise them. If your problem requires relatively high performance: Java, Go and Rust are all good alternatives. If absolute pedal-to-the-metal, no compromises, performance is needed: C, C++ are your choices. Heck, you can even inline assembly in some cases.
Areading314 6y ago
This is especially true when writing code that needs to run on many cores -- Python starts getting really cumbersome in this case, which can be avoided using the relatively elegant threading models in alternative languages.
- jashmatthews 6y ago
  Task based parallelism for a web server or background job server is really easy and that's 99% of the cases covered. Problems which _need_ full multi-threading with shared mutable state are actually pretty rare.
  - Areading314 6y ago
    Agreed -- but they do happen!