9 ms·

An Attempt to Catch Up with JIT Compilers

203 points by mfiguiere 2y ago

mannyv 2y ago
[flagged]
- mannyv 2y ago
  I wonder if you could use clang/llvm to do a super-JIT by having it recompile its IR as the program runs, taking advantage of profiling to optimize the hot paths.
  - SkiFire13 2y ago
    Profile guided optimizations are already a thing and don't need JITting your program.
    - mannyv 2y ago
      Profile guided optimization is a static operation that's done after profiling the running app - unless the state of the art has changed in the last few years.
- deleted 2y ago
  [deleted]
ajross 2y ago
This seems poorly grounded. In fact almost three decades after the release of the Java HotSpot runtime we're still waiting for even one system to produce the promised advantages. I guess consensus is that V8 has come closest? But the reality is that hand-optimized AoT builds remain the gold standard for performance work.
- deleted 2y ago
  [deleted]
- noelwelsh 2y ago
  The benchmarks I have seen show Hotspot is ahead of V8. E.g. https://stefan-marr.de/papers/oopsla-larose-et-al-ast-vs-bytecode-interpreters-in-the-age-of-meta-compilation/ https://stefan-marr.de/papers/oopsla-larose-et-al-ast-vs-byt... What makes this very complicated is that 1) language design plays a big part in performance and 2) CPUs change as well and this anecdotally seems to have more impact on interpreter than compiler performance. With regards to 1), consider optimizing Javascript. It doesn't have machine integers, so you have to do a bunch of analysis to figure when something is being used as an integer and then you can make that code fast. There are many other cases. Python is even worse in this regard. In comparison AOT compiled languages are usually designed to be fast, so they make tradeoffs that favour performance at the cost of some level of abstraction / expressivity. The JVM is somewhere in the middle, and so is its performance. With regards to 2) this paper is an example, as is https://inria.hal.science/hal-01100647/file/InterpIBr-hal.pdf https://inria.hal.science/hal-01100647/file/InterpIBr-hal.pd...
  - ajross 2y ago
    With all respect that sounds like excuse-making. I mean, yeah, Javascript and JVM and .NET are slower runtimes than C or Rust[1]. Nonetheless that's the world we live in, and if you have a performance-sensitive problem to solve you pick up rustc or g++ and not a managed runtime. If that's wrong, someone's got to actually show that it's wrong. [1] Maybe Go or Swift would be more apples-to-apples. But even then are there clear benchmarks showing Kotlin or C# beating similar AoT code? If anything the general sense of the community is that Go is faster than Java.
    - noelwelsh 2y ago
      Excuses for what? I'm not the elected representative for JIT compiled languages, sworn to defend them. There are technical reasons they tend to be slower. I was sketching some of them.
      - mrkeen 2y ago
        I think the above comments are because JIT gets so much positive press, someone wandering in from outside could be mistaken for thinking that JIT isn't coming 2nd in a two-man race with AOT. I've been around long enough to hear that Java and JIT are gonna overtake C++ any day now. The title on this article doesn't help.
- ForTheKidz 2y ago
  > I guess consensus is that V8 has come closest? V8 better than the JVM? Insanity, maybe it can come to within an order of magnitude in terms of performance.
  - edflsafoiewq 2y ago
    Comes closest to realizing the concept of a JIT that is better than AOT.
    - ForTheKidz 2y ago
      I think that's completely silly framing; you can AOT compile any code better—or at least, just as well—if you already know how you want it to perform at runtime. Any efficiency gain would necessarily need to be in the context of total productivity.
      - ajross 2y ago
        > I think that's completely silly framing It's literally the framing of the linked article though, which takes as a prior that JIT compilers are already ahead of AoT toolchains. And... they aren't!
        ForTheKidz 2y ago
        > This gives them an advantage when compared to Ahead-of-Time (AoT) compilers that must choose the code to generate once for all. I assumed they were talking about the general case, which is nearly useless to discuss. I just kind of filtered it out as internecine bickering amongst academics. The actual data are still interesting tho.
        noelwelsh 2y ago
        They are comparing Javascript JIT to Javascript AOT, to avoid the issue of language design. "The fastest contemporary JavaScript implementations use JIT compilers [27]. ... However, JIT compilers may not be desirable or simply not available in some contexts, for instance if programs are to be executed on platforms with too limited resources or if the architecture forbids dynamic code generation. Ahead of time (AoT) compilers offer a response to these situations. Hopc [25] is an AoT JavaScript-to-C compiler. Its performance is often in the same range as that of the fastest JIT compilers but its impossibility to adapt the code executed at runtime seems a handicap for some patterns and benchmarks [27]." In the context of JS it's reasonable to think that JIT may have an advantage, as the language is difficult to statically analyse.
- IshKebab 2y ago
  I agree, the "JITs can be faster because X Y Z" arguments have never turned into "JITs are actually faster". Maybe that's because JIT is almost always used in languages that were slowed in the first place, e.g. due to GC. Is there a JITing C compiler, or something like that? Would that even make sense?
  - remexre 2y ago
    Maybe the "allocate as little as possible, use sun.misc.Unsafe a lot, have lots of long-lived global arrays" style of Java programming some high-performance Java programs use would get close to being a good stand-in.
  - pjmlp 2y ago
    C++/CLI is one example, it is C++, not C, but example holds.
    - do_not_redeem 2y ago
      Now the money question: can anyone come up with a benchmark where, due to the JIT, C++/CLI runs faster than normal C++ compiled for the same CPU?
      - bjoli 2y ago
        Writing a program where a jit version is faster than the aot version is just an exercise in knowing the limitations of AOT. People have been doing runtime code generation for a very long time for exactly this reason. A general implementation faster than, say, g++ is a completely different beast.
    - zabzonk 2y ago
      It is not C++ (or C) but a Microsoft invented language - which is OK, but don't confuse it with C++ anymore than MS have already done
      - pjmlp 2y ago
- pizlonator 2y ago
  > Java HotSpot runtime we're still waiting for even one system to produce the promised advantages. What promised advantages are you waiting on? There are lots of systems that have architectures that are similar to HotSpot, or that surpass it in some way. V8 is just one.
  - CamouflagedKiwi 2y ago
    There were many many statements made that JIT compilers could be faster than AOT compilers because they had more information to use at runtime - originally this was mostly aimed at Java/HotSpot which has not, in practice, significantly displaced languages like C or C++ (or these days Rust) from high-performance work.
    - pjmlp 2y ago
      I guess distributed systems and OS GUI frameworks aren't it then.
    - pizlonator 2y ago
      Yeah those statements were overly optimistic and I don’t think they’re representative of what most people in the JIT field think. It’s also not what I as a JIT engineer would have promised you. The actual promise is just: JITs make dynamic languages faster and they are better at doing that than AOTs. I think lots of systems have delivered on that promise.
      - CamouflagedKiwi 2y ago
        Yup, agreed, in the case of dynamic languages it's much clearer and the evidence is a lot more favourable. The linked article doesn't help here because the abstract only mentions Javascript in the context of their work to prove their concept, but the body of the paper is clearer that it is discussing JIT vs AOT in the context of Javascript specifically.
        pizlonator 2y ago
- pjmlp 2y ago
  JVM implementations, especially those with PGO feedback loop across runs do quite well. Likewise modern Android, runs reasonably well with its mix of JIT, AOT with JIT PGO metadata, baseline profiles shared across devices via Play Store. The gold standard for anyone that actually cares about ultimate performance is hand written Assembly, naturally guided with a profilers capable to measure everything that the CPU is doing like VTune.
- titzer 2y ago
  > But the reality is that hand-optimized AoT builds remain the gold standard for performance work. It's considerably more complicated than that. After working in this area for 25 years, I have vacillated between extremes over decades-long arcs. The reality is much more nuanced than a four sentence HN comment. Profile and measure and stare at machine code. If you don't do that daily, it's hand waving and having hunches.
  - cogman10 2y ago
    I'd also point out that it's an ever-shifting landscape. What was slow yesterday might not be today. In my experience, while there are some negatives of the runtime selected, the vast majority of performance is won or lost at the algorithm level. It really doesn't matter that rust can be faster than ruby if you chose an O(n^3) algorithm. Rust will run the O(n^3) algorithm faster than ruby, for sure, but ruby will beat the pants off of rust if someone converts it into an O(n) algorithm. It only starts mattering if you've already have an O(n) algorithm. However, in my experience, a LOT of programmers are happy writing a n^3 and moving on to the next task without considering what this will do. for (i : foo) { for (j : foo) { for (k : foo) { bar(i, j, k) } } }
    - neonsunset 2y ago
      You may be underestimating the degree of difference in performance between Ruby and Rust. Here's comparison of Ruby with JS, and Rust is of course faster still: https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/node-ruby.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... If the code runs 100 times faster, it might just offset even highly inefficient implementation. > a LOT of programmers are happy writing a n^3 I have the same experience. Unfortunately, and this is an issue I keep fighting with in some .NET communities, languages like C, C++ and Rust tend to select for engineers which are more likely to care about writing reasonably efficient implementation. At the same time, higher-level languages sometimes can almost encourage the blindness to the real world model of computation, the execution implications be damned. In such languages you will encounter way more people who will write O(n^3) algorithm and will fight you tooth and nail to keep it that way because they have zero understanding of the fundamentals, wasting the heroic effort by the runtime/compiler to keep it running acceptably well.
- neonsunset 2y ago
  If you pit virtual-call-heavy code written in C++ against C#, C# will come out on top every single time, especially if you consume dynamically-linked dependencies or if you can't afford to wait until the heat death of the universe when all the LTO plugins finish their job. Or if you use SIMD-heavy path and your binary is built against, say, X86-64-v2/3 and the target supports AVX512, .NET will happily use the entirety of AVX512 thanks to JIT even when still using 256b-wide operations (i.e. bespoke path that uses Vector256) with AVX512VL. This tends to surpass what you can get out of runtime dispatch under LLVM. re: Java challenges - those stem from the JVM bytecode being a very difficult optimization target i.e. every call is virtual by default with complex dispatch strategy, everything is a heap-allocated object by default save for very few primitives, generics lose type information and are never monomorphized - PGO optimization through tiered compilation and resulting guarded devirtualization and object escape analysis is something that reclaims performance in Java and makes it acceptable. C and C++ with templates are a massively easier optimization target for GCC, and GCC does not operate under strict time constraints too. Therefore we have the results that we do. Also interesting data points here if you'd like to look at AOT capabilities of higher-level languages: https://benchmarksgame-team.pages.debian.net/benchmarksgame/box-plot-summary-charts.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- twoodfin 2y ago
  Hand-optimized AoT builds with solid profile-based feedback, right?
- paulddraper 2y ago
  > we're still waiting for even one system to produce the promised advantages To be clear, successful JIT do runtime profiling+optimization, at significant benefit. But on net, JIT languages are slower. It is a valid question to ask whether AOT binaries can selectively use runtime optimizations, making them even faster.
tsunego 2y ago
chasing inline cache micro-optimizations with dynamic binary modification is a dead end. modern CPUs are laughing at our outdated compiler tricks. maybe it's time to accept that clever hacks won’t outrun silicon.
- andrekandre 2y ago
  what is the better approach?
  - Sparkyte 2y ago
    You don't, there are equal trade offs. JIT might use more memory because of what it does at the runtime, but it is also the exact reason it is faster to start. A good trade off is just using the type of languages best suited for the workload.
- saagarjha 2y ago
  JITs typically are too broken for compiler tricks so I don't think it's time to accept that just yet.
pizlonator 2y ago
I think the missing piece here is that JavaScriptCore (JSC) and other such systems don't just use inline caching to speed up dynamic accesses; they use them as profiling feedback. So, anytime you have an IC in interpreter, baseline, or lightly optimized code, then that IC is monitored to see how polymorphic it gets, and that data is fed back into the optimization pipeline. Just having an IC as a dead-end, where you don't use it for profiling is way less profitable than having an IC that feeds into profiling.
- titzer 2y ago
  Indeed, this was literally the conclusion of the first paper that introduced polymorphic inline caches. I'll add that the real benefit of ICs isn't just that compiled code is specialized to the seen types, but the fact that deoptimization guards are inserted, which split diamonds in the original general cases so that multiple downstream checks become redundant. So specialization is not just a local simplification but a global simplification to all dominated code in the context of the compilation unit.
  - sitkack 2y ago
    https://bibliography.selflanguage.org/pics.html https://bibliography.selflanguage.org/pics.html for those following along
- kannanvijayan 2y ago
  Well on dynamic languages the ICs do give a nice order of magnitude speed-up by themselves, since the guard eliminates a whole hashtable (or linear) lookup instead of (in this case) a single memory indirection. But yeah - on spidermonkey we found that orienting our ICs towards being stable and easy to work with, as opposed to just being fast, ended up leading to a much better design. This is a nice result though. Negative, but good that they published it. What would be a good next step is some QEMU-style transformation, pull out basic blocks, profile them for both hotness, and incoming arguments at function starts, and dynamic dispatch targets.. then use that to re-compile the whole thing using method-jit and in particular inlining across call-paths with GVN and DCE applied. I kind of expect the results to be very positive, just based on intuition.. but it'd be cool to see how it actually turned out.
deleted 2y ago
[deleted]
VeejayRampay 2y ago
the people who came up with this are obviously brilliant but being french myself, I really wonder why no one is proof-reading the english, this gives an overall bad impression of the work imho
- indolering 2y ago
  It's a preprint.
- rhelz 2y ago
  Being a native English speaker I absolutely love reading and listening to speakers of English as a second language. Speaking is actually a subspecies of singing, and it's always cool to hear the same old lyrics remixed to a new melody and a new beat. English has no 'correct' way to be written or spoken, nor does it need one, nor would it benefit from one, therefore, nor should it have one. Speakers of English as a second language: you are what makes English a great language.
  - davidgay 2y ago
    > English has no 'correct' way to be written or spoken, nor does it need one, nor would it benefit from one, therefore, nor should it have one. There may be no 'correct' way, but there are plenty of 'incomprehensible' ways. I once encountered a research paper that had clearly [0] been translated word-for-word from French into English and made no sense until I translated it word-for-word back to French... [0]: actually it was only clear after I realised I should attempt the reverse translation ;)
    - rhelz 2y ago
      Sure, but frankly, I've heard plenty of people speaking the most flawless King's English who didn't make any sense at all. re: translated math papers: haha we've all been there. Once I had to read a bunch of 70's-era papers from Russian Mathematicians. The translators, bless their hearts, I'm sure knew everything there was to know about Dickens and Dostoevsky, but it was clear they had no clue what the math was all about :-) Oh well, Math is the universal language, right? chuckle
tonnydourado 2y ago
Tangentially, fuck yeah, negative results, just as good as positive ones
- rhelz 2y ago
  Amen. This paper is worth more than all of the fraudulent, unreproducible papers we are inundated with, put together and squared.
devit 2y ago
The paper seems to start with the bizarre assumption that AOT compilers need to "catch up" with JIT compilers and in particular that they benefit from inline caches for member lookup. But the fact is that AOT compilers are usually for well-designed languages that don't need those inline caches because the designers properly specified a type system that would guarantee a field is always stored at the same offset. They might benefit from a similar mechanism to predict branches and indirect branches (i.e. virtual/dynamic dispatch), but they already have compile-time profile-guided optimization and CPU branch predictors at runtime. Furthermore, for branches that always go in one direction except for seldom changes, there are also frameworks like the Linux kernel "alternatives" and "static key" mechanisms. So the opportunity for making things better with self-modifying code is limited to code where all those mechanisms don't work well, and the overhead of the runtime profiling is worth it. Which is probably very rare and not worth bringing it a JIT compiler for.
- deleted 2y ago
  [deleted]
- pizlonator 2y ago
  AOTs are behind JITs for dynamic languages. It’s super interesting to study how to make AOTs catch up in that space, so I’m glad that these folks made an effort and reported the results!
  - Sparkyte 2y ago
    The trade offs between them are meaningful. Also Rust ain't bad for an AOT.
c-smile 2y ago
Slightly orthogonal... In my Sciter, that uses QuickJS (no JIT), instead of JIT I've added C compiler. That means we can add not just JS modules but C modules too: import * as cmod from "./cmodule.c" Such cmodule will be compiled and executed on the fly into native code. Idea is simple each language is good for specific tasks. JS is flexible and C is performant - just use right tool that is most optimal for a task. c-modules play two major roles: FFI and number crunching code execution. Sciter uses TCC compiler and runtime. In total size of QuickJS + TCC binary bundle 500k + 220k = 720k. For the comparison: V8 is of 40mb size. https://sciter.com/c-modules-in-sciter/ https://sciter.com/c-modules-in-sciter/ https://sciter.com/here-we-go/ https://sciter.com/here-we-go/
- pjmlp 2y ago
  Even if I am not a big C fan, the idea is rather cool, it is a bit like having C++ on .NET via C++/CLI.
- vanderZwan 2y ago
  Interesting project! After clicking around on the website: > In almost 10 years, Sciter UI engine has become the secret weapon of success for some of the most prominent antivirus products on the market: Norton Antivirus and Internet Security, Comodo Internet Security, ESET Antivirus, BitDefender Antivirus, and others. What an intriguingly specific niche of customer! How come all these different anti-virus companies decided to use your platform?
  - c-smile 2y ago
    > anti-virus companies decided to use your platform? One of the reasons: AV application should look modern to give an impression that the app is adequate to modern threats. So while app backend is relatively stable, its UI shall be easily tweakable. CSS/HTML is good for that. Check this: https://sciter.com/wp-content/uploads/2018/06/n360.png https://sciter.com/wp-content/uploads/2018/06/n360.png
    - mathverse 2y ago
Voultapher 2y ago
Love to see negative results published, so so important. Please let's all go towards research procedure that enforces the submission of the hypothesis before any research is allowed to commence and includes enforced publishing regardless of results.
- djoldman 2y ago
  Huge upvote from me as well. Think of all the folks out there who have this idea and instead of searching for it, finding nothing, and implementing, now they can either move on or try to fiddle with this works' output.
- api 2y ago
  I've never heard that idea before, and it's so obvious. All science should be done this way. It kind of does happen in areas of science that are capital intensive like space, high energy physics, etc., because people hear about what is to be done before it is done, but it's not formalized. It should be, and it should be done with everything.
  - Voultapher 2y ago
    If we are talking reforms to science procedure, I'd also love to see 30% or so of the research funds locked away, to be then given to another team ideally at another university that get's access only to the original teams' publication and has the goal to reproduce the study. The vast majority of papers released don't contain enough information to actually repeat their work.
- RNGesus83 2y ago
  > Love to see negative results published, so so important. > > Please let's all go towards research procedure that enforces the submission of the hypothesis before any research is allowed to commence and includes enforced publishing regardless of results. Grounded theory? https://en.m.wikipedia.org/wiki/Grounded_theory https://en.m.wikipedia.org/wiki/Grounded_theory
  - 2y ago
ErikCorry 2y ago
It's good that they post negative results, but it's hard to know exactly why their attempt failed, and it's tempting for me to make guesses without doing any measurements, so let me fall for that temptation: They are patching inline-cache sites in an AOT binary and not seeing improvements. Only 17% of the inline-cache sites could be optimized to what they call O2 level (listing 7). Most could only be optimized to O1 level (listing 6). The only difference from the baseline (listing 5) to O1 is that they replaced: mov 0x101c(%rip), %rax # load the offset with mov 0x3, %rax # load the offset I'm not very suprised that this did not help much. The old load is probably hoisted up and loaded into a renamed register very early, and it won't miss in the cache. Basically they already have a pretty nice inline cache system at least for the monomorphic case, and messing with the exact instructions used to implement it doesn't help much. A JIT is able to do so much more, eg polymorphic cases, inlining of simple methods, and eliminating repeated checks of the same hidden class. Not to mention detecting at runtime that some unknown object is almost always an integer or a float and JITting code specialized for that. People new to virtual machines often focus on the compiler, whereas the stuff that moves the needle is often around the runtime. How tagged and typed data is represented, the GC implementation, and the object layout. Eg this paper explores an interesting new tagging technique and makes a huge difference to performance (there's some author overlap): https://www.researchgate.net/figure/The-three-representations-in-a-tagged-object-system-here-shown-on-a-little-endian_fig1_386112036 https://www.researchgate.net/figure/The-three-representation... Incidentally the assembly syntax in the "Attempt to catch up" article is a bit confusing. It looks like the IC addresses are very close to the code, like almost on the same page. Stack overflow explains it: GAS syntax for RIP-relative addressing looks like symbol + current_address (RIP), but it actually means symbol with respect to RIP. There's an inconsistency with numeric literals: [rip + 10] or AT&T 10(%rip) means 10 bytes past the end of this instruction [rip + a] or AT&T a(%rip) means to calculate a rel32 displacement to reach a, not RIP + symbol value. (The GAS manual documents this special interpretation)

203 points by mfiguiere 2y ago

mannyv 2y ago

[flagged]

mannyv 2y ago
I wonder if you could use clang/llvm to do a super-JIT by having it recompile its IR as the program runs, taking advantage of profiling to optimize the hot paths.
- SkiFire13 2y ago
  Profile guided optimizations are already a thing and don't need JITting your program.
  - mannyv 2y ago
    Profile guided optimization is a static operation that's done after profiling the running app - unless the state of the art has changed in the last few years.
deleted 2y ago
[deleted]

ajross 2y ago

This seems poorly grounded. In fact almost three decades after the release of the Java HotSpot runtime we're still waiting for even one system to produce the promised advantages. I guess consensus is that V8 has come closest? But the reality is that hand-optimized AoT builds remain the gold standard for performance work.

deleted 2y ago
[deleted]
noelwelsh 2y ago
The benchmarks I have seen show Hotspot is ahead of V8. E.g. https://stefan-marr.de/papers/oopsla-larose-et-al-ast-vs-bytecode-interpreters-in-the-age-of-meta-compilation/ https://stefan-marr.de/papers/oopsla-larose-et-al-ast-vs-byt... What makes this very complicated is that 1) language design plays a big part in performance and 2) CPUs change as well and this anecdotally seems to have more impact on interpreter than compiler performance. With regards to 1), consider optimizing Javascript. It doesn't have machine integers, so you have to do a bunch of analysis to figure when something is being used as an integer and then you can make that code fast. There are many other cases. Python is even worse in this regard. In comparison AOT compiled languages are usually designed to be fast, so they make tradeoffs that favour performance at the cost of some level of abstraction / expressivity. The JVM is somewhere in the middle, and so is its performance. With regards to 2) this paper is an example, as is https://inria.hal.science/hal-01100647/file/InterpIBr-hal.pdf https://inria.hal.science/hal-01100647/file/InterpIBr-hal.pd...
- ajross 2y ago
  With all respect that sounds like excuse-making. I mean, yeah, Javascript and JVM and .NET are slower runtimes than C or Rust[1]. Nonetheless that's the world we live in, and if you have a performance-sensitive problem to solve you pick up rustc or g++ and not a managed runtime. If that's wrong, someone's got to actually show that it's wrong. [1] Maybe Go or Swift would be more apples-to-apples. But even then are there clear benchmarks showing Kotlin or C# beating similar AoT code? If anything the general sense of the community is that Go is faster than Java.
  - noelwelsh 2y ago
    Excuses for what? I'm not the elected representative for JIT compiled languages, sworn to defend them. There are technical reasons they tend to be slower. I was sketching some of them.
    - mrkeen 2y ago
      I think the above comments are because JIT gets so much positive press, someone wandering in from outside could be mistaken for thinking that JIT isn't coming 2nd in a two-man race with AOT. I've been around long enough to hear that Java and JIT are gonna overtake C++ any day now. The title on this article doesn't help.
ForTheKidz 2y ago
> I guess consensus is that V8 has come closest? V8 better than the JVM? Insanity, maybe it can come to within an order of magnitude in terms of performance.
- edflsafoiewq 2y ago
  Comes closest to realizing the concept of a JIT that is better than AOT.
  - ForTheKidz 2y ago
    I think that's completely silly framing; you can AOT compile any code better—or at least, just as well—if you already know how you want it to perform at runtime. Any efficiency gain would necessarily need to be in the context of total productivity.
    - ajross 2y ago
      > I think that's completely silly framing It's literally the framing of the linked article though, which takes as a prior that JIT compilers are already ahead of AoT toolchains. And... they aren't!
      - ForTheKidz 2y ago
        > This gives them an advantage when compared to Ahead-of-Time (AoT) compilers that must choose the code to generate once for all. I assumed they were talking about the general case, which is nearly useless to discuss. I just kind of filtered it out as internecine bickering amongst academics. The actual data are still interesting tho.
      - noelwelsh 2y ago
        They are comparing Javascript JIT to Javascript AOT, to avoid the issue of language design. "The fastest contemporary JavaScript implementations use JIT compilers [27]. ... However, JIT compilers may not be desirable or simply not available in some contexts, for instance if programs are to be executed on platforms with too limited resources or if the architecture forbids dynamic code generation. Ahead of time (AoT) compilers offer a response to these situations. Hopc [25] is an AoT JavaScript-to-C compiler. Its performance is often in the same range as that of the fastest JIT compilers but its impossibility to adapt the code executed at runtime seems a handicap for some patterns and benchmarks [27]." In the context of JS it's reasonable to think that JIT may have an advantage, as the language is difficult to statically analyse.
IshKebab 2y ago
I agree, the "JITs can be faster because X Y Z" arguments have never turned into "JITs are actually faster". Maybe that's because JIT is almost always used in languages that were slowed in the first place, e.g. due to GC. Is there a JITing C compiler, or something like that? Would that even make sense?
- remexre 2y ago
  Maybe the "allocate as little as possible, use sun.misc.Unsafe a lot, have lots of long-lived global arrays" style of Java programming some high-performance Java programs use would get close to being a good stand-in.
- pjmlp 2y ago
  C++/CLI is one example, it is C++, not C, but example holds.
  - do_not_redeem 2y ago
    Now the money question: can anyone come up with a benchmark where, due to the JIT, C++/CLI runs faster than normal C++ compiled for the same CPU?
    - bjoli 2y ago
      Writing a program where a jit version is faster than the aot version is just an exercise in knowing the limitations of AOT. People have been doing runtime code generation for a very long time for exactly this reason. A general implementation faster than, say, g++ is a completely different beast.
  - zabzonk 2y ago
    It is not C++ (or C) but a Microsoft invented language - which is OK, but don't confuse it with C++ anymore than MS have already done
    - pjmlp 2y ago
pizlonator 2y ago
> Java HotSpot runtime we're still waiting for even one system to produce the promised advantages. What promised advantages are you waiting on? There are lots of systems that have architectures that are similar to HotSpot, or that surpass it in some way. V8 is just one.
- CamouflagedKiwi 2y ago
  There were many many statements made that JIT compilers could be faster than AOT compilers because they had more information to use at runtime - originally this was mostly aimed at Java/HotSpot which has not, in practice, significantly displaced languages like C or C++ (or these days Rust) from high-performance work.
  - pjmlp 2y ago
    I guess distributed systems and OS GUI frameworks aren't it then.
  - pizlonator 2y ago
    Yeah those statements were overly optimistic and I don’t think they’re representative of what most people in the JIT field think. It’s also not what I as a JIT engineer would have promised you. The actual promise is just: JITs make dynamic languages faster and they are better at doing that than AOTs. I think lots of systems have delivered on that promise.
    - CamouflagedKiwi 2y ago
      Yup, agreed, in the case of dynamic languages it's much clearer and the evidence is a lot more favourable. The linked article doesn't help here because the abstract only mentions Javascript in the context of their work to prove their concept, but the body of the paper is clearer that it is discussing JIT vs AOT in the context of Javascript specifically.
      - pizlonator 2y ago
pjmlp 2y ago
JVM implementations, especially those with PGO feedback loop across runs do quite well. Likewise modern Android, runs reasonably well with its mix of JIT, AOT with JIT PGO metadata, baseline profiles shared across devices via Play Store. The gold standard for anyone that actually cares about ultimate performance is hand written Assembly, naturally guided with a profilers capable to measure everything that the CPU is doing like VTune.
titzer 2y ago
> But the reality is that hand-optimized AoT builds remain the gold standard for performance work. It's considerably more complicated than that. After working in this area for 25 years, I have vacillated between extremes over decades-long arcs. The reality is much more nuanced than a four sentence HN comment. Profile and measure and stare at machine code. If you don't do that daily, it's hand waving and having hunches.
- cogman10 2y ago
  I'd also point out that it's an ever-shifting landscape. What was slow yesterday might not be today. In my experience, while there are some negatives of the runtime selected, the vast majority of performance is won or lost at the algorithm level. It really doesn't matter that rust can be faster than ruby if you chose an O(n^3) algorithm. Rust will run the O(n^3) algorithm faster than ruby, for sure, but ruby will beat the pants off of rust if someone converts it into an O(n) algorithm. It only starts mattering if you've already have an O(n) algorithm. However, in my experience, a LOT of programmers are happy writing a n^3 and moving on to the next task without considering what this will do. for (i : foo) { for (j : foo) { for (k : foo) { bar(i, j, k) } } }
  - neonsunset 2y ago
    You may be underestimating the degree of difference in performance between Ruby and Rust. Here's comparison of Ruby with JS, and Rust is of course faster still: https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/node-ruby.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... If the code runs 100 times faster, it might just offset even highly inefficient implementation. > a LOT of programmers are happy writing a n^3 I have the same experience. Unfortunately, and this is an issue I keep fighting with in some .NET communities, languages like C, C++ and Rust tend to select for engineers which are more likely to care about writing reasonably efficient implementation. At the same time, higher-level languages sometimes can almost encourage the blindness to the real world model of computation, the execution implications be damned. In such languages you will encounter way more people who will write O(n^3) algorithm and will fight you tooth and nail to keep it that way because they have zero understanding of the fundamentals, wasting the heroic effort by the runtime/compiler to keep it running acceptably well.
neonsunset 2y ago
If you pit virtual-call-heavy code written in C++ against C#, C# will come out on top every single time, especially if you consume dynamically-linked dependencies or if you can't afford to wait until the heat death of the universe when all the LTO plugins finish their job. Or if you use SIMD-heavy path and your binary is built against, say, X86-64-v2/3 and the target supports AVX512, .NET will happily use the entirety of AVX512 thanks to JIT even when still using 256b-wide operations (i.e. bespoke path that uses Vector256) with AVX512VL. This tends to surpass what you can get out of runtime dispatch under LLVM. re: Java challenges - those stem from the JVM bytecode being a very difficult optimization target i.e. every call is virtual by default with complex dispatch strategy, everything is a heap-allocated object by default save for very few primitives, generics lose type information and are never monomorphized - PGO optimization through tiered compilation and resulting guarded devirtualization and object escape analysis is something that reclaims performance in Java and makes it acceptable. C and C++ with templates are a massively easier optimization target for GCC, and GCC does not operate under strict time constraints too. Therefore we have the results that we do. Also interesting data points here if you'd like to look at AOT capabilities of higher-level languages: https://benchmarksgame-team.pages.debian.net/benchmarksgame/box-plot-summary-charts.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
twoodfin 2y ago
Hand-optimized AoT builds with solid profile-based feedback, right?
paulddraper 2y ago
> we're still waiting for even one system to produce the promised advantages To be clear, successful JIT do runtime profiling+optimization, at significant benefit. But on net, JIT languages are slower. It is a valid question to ask whether AOT binaries can selectively use runtime optimizations, making them even faster.

tsunego 2y ago

chasing inline cache micro-optimizations with dynamic binary modification is a dead end. modern CPUs are laughing at our outdated compiler tricks. maybe it's time to accept that clever hacks won’t outrun silicon.

andrekandre 2y ago
what is the better approach?
- Sparkyte 2y ago
  You don't, there are equal trade offs. JIT might use more memory because of what it does at the runtime, but it is also the exact reason it is faster to start. A good trade off is just using the type of languages best suited for the workload.
saagarjha 2y ago
JITs typically are too broken for compiler tricks so I don't think it's time to accept that just yet.

pizlonator 2y ago

I think the missing piece here is that JavaScriptCore (JSC) and other such systems don't just use inline caching to speed up dynamic accesses; they use them as profiling feedback. So, anytime you have an IC in interpreter, baseline, or lightly optimized code, then that IC is monitored to see how polymorphic it gets, and that data is fed back into the optimization pipeline. Just having an IC as a dead-end, where you don't use it for profiling is way less profitable than having an IC that feeds into profiling.

titzer 2y ago
Indeed, this was literally the conclusion of the first paper that introduced polymorphic inline caches. I'll add that the real benefit of ICs isn't just that compiled code is specialized to the seen types, but the fact that deoptimization guards are inserted, which split diamonds in the original general cases so that multiple downstream checks become redundant. So specialization is not just a local simplification but a global simplification to all dominated code in the context of the compilation unit.
- sitkack 2y ago
  https://bibliography.selflanguage.org/pics.html https://bibliography.selflanguage.org/pics.html for those following along
kannanvijayan 2y ago
Well on dynamic languages the ICs do give a nice order of magnitude speed-up by themselves, since the guard eliminates a whole hashtable (or linear) lookup instead of (in this case) a single memory indirection. But yeah - on spidermonkey we found that orienting our ICs towards being stable and easy to work with, as opposed to just being fast, ended up leading to a much better design. This is a nice result though. Negative, but good that they published it. What would be a good next step is some QEMU-style transformation, pull out basic blocks, profile them for both hotness, and incoming arguments at function starts, and dynamic dispatch targets.. then use that to re-compile the whole thing using method-jit and in particular inlining across call-paths with GVN and DCE applied. I kind of expect the results to be very positive, just based on intuition.. but it'd be cool to see how it actually turned out.

deleted 2y ago

[deleted]

VeejayRampay 2y ago

the people who came up with this are obviously brilliant but being french myself, I really wonder why no one is proof-reading the english, this gives an overall bad impression of the work imho

indolering 2y ago
It's a preprint.
rhelz 2y ago
Being a native English speaker I absolutely love reading and listening to speakers of English as a second language. Speaking is actually a subspecies of singing, and it's always cool to hear the same old lyrics remixed to a new melody and a new beat. English has no 'correct' way to be written or spoken, nor does it need one, nor would it benefit from one, therefore, nor should it have one. Speakers of English as a second language: you are what makes English a great language.
- davidgay 2y ago
  > English has no 'correct' way to be written or spoken, nor does it need one, nor would it benefit from one, therefore, nor should it have one. There may be no 'correct' way, but there are plenty of 'incomprehensible' ways. I once encountered a research paper that had clearly [0] been translated word-for-word from French into English and made no sense until I translated it word-for-word back to French... [0]: actually it was only clear after I realised I should attempt the reverse translation ;)
  - rhelz 2y ago
    Sure, but frankly, I've heard plenty of people speaking the most flawless King's English who didn't make any sense at all. re: translated math papers: haha we've all been there. Once I had to read a bunch of 70's-era papers from Russian Mathematicians. The translators, bless their hearts, I'm sure knew everything there was to know about Dickens and Dostoevsky, but it was clear they had no clue what the math was all about :-) Oh well, Math is the universal language, right? chuckle

tonnydourado 2y ago

Tangentially, fuck yeah, negative results, just as good as positive ones

rhelz 2y ago
Amen. This paper is worth more than all of the fraudulent, unreproducible papers we are inundated with, put together and squared.

devit 2y ago

The paper seems to start with the bizarre assumption that AOT compilers need to "catch up" with JIT compilers and in particular that they benefit from inline caches for member lookup. But the fact is that AOT compilers are usually for well-designed languages that don't need those inline caches because the designers properly specified a type system that would guarantee a field is always stored at the same offset. They might benefit from a similar mechanism to predict branches and indirect branches (i.e. virtual/dynamic dispatch), but they already have compile-time profile-guided optimization and CPU branch predictors at runtime. Furthermore, for branches that always go in one direction except for seldom changes, there are also frameworks like the Linux kernel "alternatives" and "static key" mechanisms. So the opportunity for making things better with self-modifying code is limited to code where all those mechanisms don't work well, and the overhead of the runtime profiling is worth it. Which is probably very rare and not worth bringing it a JIT compiler for.

deleted 2y ago
[deleted]
pizlonator 2y ago
AOTs are behind JITs for dynamic languages. It’s super interesting to study how to make AOTs catch up in that space, so I’m glad that these folks made an effort and reported the results!
- Sparkyte 2y ago
  The trade offs between them are meaningful. Also Rust ain't bad for an AOT.

c-smile 2y ago

Slightly orthogonal... In my Sciter, that uses QuickJS (no JIT), instead of JIT I've added C compiler. That means we can add not just JS modules but C modules too: import * as cmod from "./cmodule.c" Such cmodule will be compiled and executed on the fly into native code. Idea is simple each language is good for specific tasks. JS is flexible and C is performant - just use right tool that is most optimal for a task. c-modules play two major roles: FFI and number crunching code execution. Sciter uses TCC compiler and runtime. In total size of QuickJS + TCC binary bundle 500k + 220k = 720k. For the comparison: V8 is of 40mb size. https://sciter.com/c-modules-in-sciter/ https://sciter.com/c-modules-in-sciter/ https://sciter.com/here-we-go/ https://sciter.com/here-we-go/

pjmlp 2y ago
Even if I am not a big C fan, the idea is rather cool, it is a bit like having C++ on .NET via C++/CLI.
vanderZwan 2y ago
Interesting project! After clicking around on the website: > In almost 10 years, Sciter UI engine has become the secret weapon of success for some of the most prominent antivirus products on the market: Norton Antivirus and Internet Security, Comodo Internet Security, ESET Antivirus, BitDefender Antivirus, and others. What an intriguingly specific niche of customer! How come all these different anti-virus companies decided to use your platform?
- c-smile 2y ago
  > anti-virus companies decided to use your platform? One of the reasons: AV application should look modern to give an impression that the app is adequate to modern threats. So while app backend is relatively stable, its UI shall be easily tweakable. CSS/HTML is good for that. Check this: https://sciter.com/wp-content/uploads/2018/06/n360.png https://sciter.com/wp-content/uploads/2018/06/n360.png
  - mathverse 2y ago

Voultapher 2y ago

Love to see negative results published, so so important. Please let's all go towards research procedure that enforces the submission of the hypothesis before any research is allowed to commence and includes enforced publishing regardless of results.

djoldman 2y ago
Huge upvote from me as well. Think of all the folks out there who have this idea and instead of searching for it, finding nothing, and implementing, now they can either move on or try to fiddle with this works' output.
api 2y ago
I've never heard that idea before, and it's so obvious. All science should be done this way. It kind of does happen in areas of science that are capital intensive like space, high energy physics, etc., because people hear about what is to be done before it is done, but it's not formalized. It should be, and it should be done with everything.
- Voultapher 2y ago
  If we are talking reforms to science procedure, I'd also love to see 30% or so of the research funds locked away, to be then given to another team ideally at another university that get's access only to the original teams' publication and has the goal to reproduce the study. The vast majority of papers released don't contain enough information to actually repeat their work.
RNGesus83 2y ago
> Love to see negative results published, so so important. > > Please let's all go towards research procedure that enforces the submission of the hypothesis before any research is allowed to commence and includes enforced publishing regardless of results. Grounded theory? https://en.m.wikipedia.org/wiki/Grounded_theory https://en.m.wikipedia.org/wiki/Grounded_theory
- 2y ago

ErikCorry 2y ago

It's good that they post negative results, but it's hard to know exactly why their attempt failed, and it's tempting for me to make guesses without doing any measurements, so let me fall for that temptation: They are patching inline-cache sites in an AOT binary and not seeing improvements. Only 17% of the inline-cache sites could be optimized to what they call O2 level (listing 7). Most could only be optimized to O1 level (listing 6). The only difference from the baseline (listing 5) to O1 is that they replaced: mov 0x101c(%rip), %rax # load the offset with mov 0x3, %rax # load the offset I'm not very suprised that this did not help much. The old load is probably hoisted up and loaded into a renamed register very early, and it won't miss in the cache. Basically they already have a pretty nice inline cache system at least for the monomorphic case, and messing with the exact instructions used to implement it doesn't help much. A JIT is able to do so much more, eg polymorphic cases, inlining of simple methods, and eliminating repeated checks of the same hidden class. Not to mention detecting at runtime that some unknown object is almost always an integer or a float and JITting code specialized for that. People new to virtual machines often focus on the compiler, whereas the stuff that moves the needle is often around the runtime. How tagged and typed data is represented, the GC implementation, and the object layout. Eg this paper explores an interesting new tagging technique and makes a huge difference to performance (there's some author overlap): https://www.researchgate.net/figure/The-three-representations-in-a-tagged-object-system-here-shown-on-a-little-endian_fig1_386112036 https://www.researchgate.net/figure/The-three-representation... Incidentally the assembly syntax in the "Attempt to catch up" article is a bit confusing. It looks like the IC addresses are very close to the code, like almost on the same page. Stack overflow explains it: GAS syntax for RIP-relative addressing looks like symbol + current_address (RIP), but it actually means symbol with respect to RIP. There's an inconsistency with numeric literals: [rip + 10] or AT&T 10(%rip) means 10 bytes past the end of this instruction [rip + a] or AT&T a(%rip) means to calculate a rel32 displacement to reach a, not RIP + symbol value. (The GAS manual documents this special interpretation)