3 ms·

> Theoretically, a JIT could produce binary code hyper-tailored to a particular user's habits and their computer's specific hardware. However, I'm not sure if t

> Theoretically, a JIT could produce binary code hyper-tailored to a particular user's habits and their computer's specific hardware. However, I'm not sure if that has that much of a benefit versus PGO AOT. In theory JIT can be a lot more efficient, optimizing for not only the exact instruction set, and do per CPU architecture optimizations, such as instruction length, pipeline depth, cache sizes, etc. In reality I doubt most compiler or JIT development teams have the resources to write and test all those potential optimizations, especially as new CPUs are coming out all the time, and each set of optimizations is another set of tests that has to be maintained.

saagarjha 4y ago
The well funded production JIT compilers (HotSpot, V8, etc.) absolutely do take advantage of these. The vector ISA can sometimes be unwieldy to work with but things like replacing atomics, using unaligned loads, or taking advantage of differing pointer representations is common.
- com2kid 4y ago
  They do some auto-vectorization, but AFAIK they don't do micro-optimizations for different CPUs.
duped 4y ago
Like another commented, JIT compilers do this today. The thing that makes this mostly theoretical is that the underlying assumption is only true when you neglect that an AOT has zero run-time cost while a JIT compiler has to execute the code it's optimizing and the code to decide if it's worth optimizing and generate new code. So JIT compiler optimizations are a bit different than AOT optimizations since they have to both generate faster/smaller code and the execute code that performs the optimization. The problem is that most optimizations beyond peephole are quite expensive. There's another thing that AOT compilers don't need to deal with, which is being wrong.Production JITs have to implement dynamic de-optimization in the case that an optimization was built on a bad assumption. That's why JITs are only faster in theory (today), since there are performance pitfalls in the JIT itself.
- KMag 4y ago
  But, JIT vs. AoT is a false dichotomy. Given light-weight enough profiling utilizing cooperation between hardware designers and compiler writers, one could have AoT with feedback-guided optimization and link-time optimization, and still have just-in-time re-optimization. Concretely, I think you'd want hardware that supported reservoir sampling of where CPU cycles are spent, sampling of which branches are mispredicted, and which code locations are causing cache misses. You'd also want lightweight hardware recording of execution traces.
  - saagarjha 4y ago
    Guess what Apple has? :)
- titzer 4y ago
  Nearly all JS engines are doing concurrent JIT compilation now, so some of the compilation cost is moved off the main thread. Java JITs have had multiple compiler threads for more than a decade.
  - kllrnohj 4y ago
    But they all still optimize their JITs to prioritize compilation speed & RAM usage (JIT'd code is dirty pages after all) over maximum optimizations. This is why you see things like WebKits multi-tier JIT strategy: https://webkit.org/blog/3362/introducing-the-webkit-ftl-jit/ https://webkit.org/blog/3362/introducing-the-webkit-ftl-jit/ They still want to swap in that JIT'd result ASAP since after all by the time it's been flagged for compilation it's already too late & is a hot hot hot function.
    - pjmlp 4y ago
      Which is why in the Java world (including Android), and .NET, now we use JIT caches as well, so that this data isn't lost between runs.
bluGill 4y ago
gcc and clang at least have options so you can optimize for specific CPUs. I'm not sure how good they are (most people want a generic optimization that runs well on all CPUs of the family, so there likely is lots of room for improvement with CPU specific optimization), but they can do that. This does (or at least can, again it probably isn't fully implemented), account for instruction length, pipeline depth, cache size. The Javascript V8 engine, and the JVM both are popular and supported enough that I expect the teams working on them take advantage of every trick they can for specific CPUs, they have a lot of resources for this. (at least the major x86 and ARM chips - maybe they don't for MIPS or some uncommon variant of ARM...). Of courses there are other JIT engines, some uncommon ones don't have many resources and won't do this.
- titzer 4y ago
  > take advantage of every trick they can for specific CPUs Not to the extent clang and gcc do, no. V8 does, e.g. use AVX instructions and some others if they are indicated to be available by CPUID. TurboFan does global scheduling in moving out of the sea of nodes, but that is not machine-specific. There was an experimental local instruction scheduler for TurboFan but it never really helped big cores, while measurements showed it would have helped smaller cores. It didn't actually calculate latencies; it just used a greedy heuristic. I am not sure if it was ever turned on. TurboFan doesn't do software pipelining or unroll/jam, though it does loop peeling, which isn't CPU-specific.
- astrange 4y ago
  > gcc and clang at least have options so you can optimize for specific CPUs. I'm not sure how good they are They are not very good at it, and can't be. You can look inside them and see the models are pretty simple; the best you can do is optimize for the first step (decoder) of the CPU and avoid instructions called out in the optimization manual as being especially slow. But on an OoO CPU there's not much else you can do ahead of time, since branches and memory accesses are unpredictable and much slower than in-CPU resource stalls.

saagarjha 4y ago

The well funded production JIT compilers (HotSpot, V8, etc.) absolutely do take advantage of these. The vector ISA can sometimes be unwieldy to work with but things like replacing atomics, using unaligned loads, or taking advantage of differing pointer representations is common.

com2kid 4y ago
They do some auto-vectorization, but AFAIK they don't do micro-optimizations for different CPUs.

duped 4y ago

Like another commented, JIT compilers do this today. The thing that makes this mostly theoretical is that the underlying assumption is only true when you neglect that an AOT has zero run-time cost while a JIT compiler has to execute the code it's optimizing and the code to decide if it's worth optimizing and generate new code. So JIT compiler optimizations are a bit different than AOT optimizations since they have to both generate faster/smaller code and the execute code that performs the optimization. The problem is that most optimizations beyond peephole are quite expensive. There's another thing that AOT compilers don't need to deal with, which is being wrong.Production JITs have to implement dynamic de-optimization in the case that an optimization was built on a bad assumption. That's why JITs are only faster in theory (today), since there are performance pitfalls in the JIT itself.

KMag 4y ago
But, JIT vs. AoT is a false dichotomy. Given light-weight enough profiling utilizing cooperation between hardware designers and compiler writers, one could have AoT with feedback-guided optimization and link-time optimization, and still have just-in-time re-optimization. Concretely, I think you'd want hardware that supported reservoir sampling of where CPU cycles are spent, sampling of which branches are mispredicted, and which code locations are causing cache misses. You'd also want lightweight hardware recording of execution traces.
- saagarjha 4y ago
  Guess what Apple has? :)
titzer 4y ago
Nearly all JS engines are doing concurrent JIT compilation now, so some of the compilation cost is moved off the main thread. Java JITs have had multiple compiler threads for more than a decade.
- kllrnohj 4y ago
  But they all still optimize their JITs to prioritize compilation speed & RAM usage (JIT'd code is dirty pages after all) over maximum optimizations. This is why you see things like WebKits multi-tier JIT strategy: https://webkit.org/blog/3362/introducing-the-webkit-ftl-jit/ https://webkit.org/blog/3362/introducing-the-webkit-ftl-jit/ They still want to swap in that JIT'd result ASAP since after all by the time it's been flagged for compilation it's already too late & is a hot hot hot function.
  - pjmlp 4y ago
    Which is why in the Java world (including Android), and .NET, now we use JIT caches as well, so that this data isn't lost between runs.

bluGill 4y ago

gcc and clang at least have options so you can optimize for specific CPUs. I'm not sure how good they are (most people want a generic optimization that runs well on all CPUs of the family, so there likely is lots of room for improvement with CPU specific optimization), but they can do that. This does (or at least can, again it probably isn't fully implemented), account for instruction length, pipeline depth, cache size. The Javascript V8 engine, and the JVM both are popular and supported enough that I expect the teams working on them take advantage of every trick they can for specific CPUs, they have a lot of resources for this. (at least the major x86 and ARM chips - maybe they don't for MIPS or some uncommon variant of ARM...). Of courses there are other JIT engines, some uncommon ones don't have many resources and won't do this.

titzer 4y ago
> take advantage of every trick they can for specific CPUs Not to the extent clang and gcc do, no. V8 does, e.g. use AVX instructions and some others if they are indicated to be available by CPUID. TurboFan does global scheduling in moving out of the sea of nodes, but that is not machine-specific. There was an experimental local instruction scheduler for TurboFan but it never really helped big cores, while measurements showed it would have helped smaller cores. It didn't actually calculate latencies; it just used a greedy heuristic. I am not sure if it was ever turned on. TurboFan doesn't do software pipelining or unroll/jam, though it does loop peeling, which isn't CPU-specific.
astrange 4y ago
> gcc and clang at least have options so you can optimize for specific CPUs. I'm not sure how good they are They are not very good at it, and can't be. You can look inside them and see the models are pretty simple; the best you can do is optimize for the first step (decoder) of the CPU and avoid instructions called out in the optimization manual as being especially slow. But on an OoO CPU there's not much else you can do ahead of time, since branches and memory accesses are unpredictable and much slower than in-CPU resource stalls.