4 ms·

I'm still not decided on AOT vs JIT being the endgame. In theory JIT should be higher performance, because it benefits from statistics taken at actual runtime.

I'm still not decided on AOT vs JIT being the endgame. In theory JIT should be higher performance, because it benefits from statistics taken at actual runtime. Given a smart enough compiler. But as a piece of code matures and gets more stable, the envelope of executions is better known and programmers can encode that at compile-time. That's the tradeoff taken by Rust: ask for more proofs from the programmers, and Rust is continuing to pick up speed. That's also what the Leyden project / condensers [1] is about, if I understand correctly. Pick up proofs and guarantees as early as possible and transform the program. For example by constant-propagating a configuration file taken up during build-time. Something I've pondered over the years: a programmer's job is not to produce code. It is to produce proofs and guarantees (yet another digression/rant: generating code was never a problem. Before LLMs we could copy-paste code from StackOverflow just fine) In the end it's only about marginal improvements though. These could be superseded by changes of paradigm like RAM getting some compute capabilities; or programs being split into a myriad of specialized instructions. For example filters, rules and parsing going inside the network card; SQL projections and filters going into the SSD controller; or matrix-multiplication going into integrated GPU/TPU/etc just like now. [1] https://openjdk.org/projects/leyden/notes/03-toward-condensers https://openjdk.org/projects/leyden/notes/03-toward-condense...

pjmlp 3y ago
The best solution isn't AOT vs JIT, rather JIT and AOT, having both available as standard part of the tooling. Android has learnt to have both, and thanks to PGO being shared across devices via Play Store, the AOT/JIT outcome reaches the ideal optimum for a specific application. Azul and IBM have similar approaches on their JVMs with a cluster based JIT, and JIT caches as AOT alternative. Also stuff like GPGPU is a mix of AOT and JIT, and is doing quite alright. I am not so confident with LLMs, when they get good enough programmers will be left out of the loop, and will have to contend to similar roles as when doing no-code SaaS configs or some form of architects. A few programmers will remain as the LLMs high priests.
- BenoitP 3y ago
  > A few programmers will remain as the LLMs high priests. That's interesting. It's controversial to say that in 2024, but not all opinions have the same value. Some are great, but some are plain dumb. The current corporate right opinion is to praise LLMs as end-all be-all. I've been asked to advise a private banking family office wanting to get into LLMs. For advising their clients' financial decisions. I politely declined. Can there be a worse use case? LLMs are parrots with the brain size of the internet. With thoughts of random origin mixed together randomly. It produces wonderful form, but abysmal analysis. IMHO as LLMs will begin to be indistinguishable from real users (and internet dogs), there's going to be a resurging need to trace origin to a human; and maybe to also rank their opinions as well. My money is on some form of distributed social proof designating the high priests.
  - pjmlp 3y ago
    I see the current state of LLMs are when we read about Assembly programmers being suspicious something like high level languages would ever take off. When we read about history of Fortran, there are several remarks on the amount of work put into place to win over those developers, as otherwise Fortran would have been yet another failed attempt. LLMs seem to be at a similar stage, maybe their Fortran moment isn't yet here, parrots as you say, but it will come.
tenaf0 3y ago
I do think, that in the general case, a JIT compiler is required: you can’t make every program fast, without having the ability to synthesize new code based on only-runtime available information. There are many where AOT is more than enough, but not all are such. Note, this doesn’t preclude AOT/hybrid models as pjmlp correctly says. One stereotypical (but not the best) example would be regexes: you basically want to compile some AST into a mini-program. This can also be done with a tiny interpreter without JIT, which will be quite competitive in speed (I believe that’s what rust has, and it’s indeed one of the fastest - the advantage of the problem/domain here is that you really can have tiny interpreters that efficiently use the caches, having very little overhead on today’s CPUs), but I am quite sure that a “JITted rust” with all the other optimizations/memory layouts could potentially fair better, but of course it’s not a trivial additional complexity.
funcDropShadow 3y ago
I am sure there will never be the conclusion that AOT is always better or that JIT is always better. E.g. JIT has a large advantage in long running applications that run at different customers with very different configurations. The scenarios where AOT has an advantage are well known, so I won't iterate them.

pjmlp 3y ago

The best solution isn't AOT vs JIT, rather JIT and AOT, having both available as standard part of the tooling. Android has learnt to have both, and thanks to PGO being shared across devices via Play Store, the AOT/JIT outcome reaches the ideal optimum for a specific application. Azul and IBM have similar approaches on their JVMs with a cluster based JIT, and JIT caches as AOT alternative. Also stuff like GPGPU is a mix of AOT and JIT, and is doing quite alright. I am not so confident with LLMs, when they get good enough programmers will be left out of the loop, and will have to contend to similar roles as when doing no-code SaaS configs or some form of architects. A few programmers will remain as the LLMs high priests.

BenoitP 3y ago
> A few programmers will remain as the LLMs high priests. That's interesting. It's controversial to say that in 2024, but not all opinions have the same value. Some are great, but some are plain dumb. The current corporate right opinion is to praise LLMs as end-all be-all. I've been asked to advise a private banking family office wanting to get into LLMs. For advising their clients' financial decisions. I politely declined. Can there be a worse use case? LLMs are parrots with the brain size of the internet. With thoughts of random origin mixed together randomly. It produces wonderful form, but abysmal analysis. IMHO as LLMs will begin to be indistinguishable from real users (and internet dogs), there's going to be a resurging need to trace origin to a human; and maybe to also rank their opinions as well. My money is on some form of distributed social proof designating the high priests.
- pjmlp 3y ago
  I see the current state of LLMs are when we read about Assembly programmers being suspicious something like high level languages would ever take off. When we read about history of Fortran, there are several remarks on the amount of work put into place to win over those developers, as otherwise Fortran would have been yet another failed attempt. LLMs seem to be at a similar stage, maybe their Fortran moment isn't yet here, parrots as you say, but it will come.

tenaf0 3y ago

I do think, that in the general case, a JIT compiler is required: you can’t make every program fast, without having the ability to synthesize new code based on only-runtime available information. There are many where AOT is more than enough, but not all are such. Note, this doesn’t preclude AOT/hybrid models as pjmlp correctly says. One stereotypical (but not the best) example would be regexes: you basically want to compile some AST into a mini-program. This can also be done with a tiny interpreter without JIT, which will be quite competitive in speed (I believe that’s what rust has, and it’s indeed one of the fastest - the advantage of the problem/domain here is that you really can have tiny interpreters that efficiently use the caches, having very little overhead on today’s CPUs), but I am quite sure that a “JITted rust” with all the other optimizations/memory layouts could potentially fair better, but of course it’s not a trivial additional complexity.

funcDropShadow 3y ago

I am sure there will never be the conclusion that AOT is always better or that JIT is always better. E.g. JIT has a large advantage in long running applications that run at different customers with very different configurations. The scenarios where AOT has an advantage are well known, so I won't iterate them.