4 ms·

What are your thoughts on whole-program optimizing compilers on this? Because, from what I've seen, with strong typing, the compiler can go to town and optimiz

What are your thoughts on whole-program optimizing compilers on this? Because, from what I've seen, with strong typing, the compiler can go to town and optimize everything, and makes JIT seem like a joke. Because, users almost always run whole programs. Why shouldn't our tools optimize for that? For the record, I like all of C++, Rust, Java, and MLton. You just need to know what you're working with (hint: it's the compiler/runtime, not the language spec)

denormalfloat 7y ago
Whole program optimization is good if your inputs never change, and you have a infrastructure to benchmark your build and recompile with profiling on each release. JITs do this automatically. A team at work had this release process. 1. Cut the branch at around midnight 2. Run the build with a portion of production traffic for a few hours 3. Collect the profiling info and feed it back into the build. 4. Repush the binary to be tested again. 5. In the morning, the team would manually push the binary to prod. The benefit was clear: about 20% reduction in CPU. However, to get to this level of automation is not easy, and you get it out of the box with JITs. One other thing: It's easy to become dependent on such performance gains. The team that had the process got into a difficult situation where there was a bug in one of the releases, couldn't roll back, and had to cherry pick a change. A few lines of code change had to go through the whole push-profile-rebuild-test cycle before it could be rolled out. Pushing the non-profiled change would have caused several other latency SLO to be violated and pagers fired. Instead, they had to wait the several hours with the bug in place, stressing over how soon the profiling would be done.
- wolf550e 7y ago
  Better PGO tooling can use the profiles from previous version of the code, which is almost but not quite the same, to compile a PGO optimized build of the patched version. If there is no tooling to do that, a subset of the training data can be used which can be processed in a short amount of time to gather enough profile data to get most of the benefits. So say, instead of 20% faster code after 6 hours, 10% faster code after 15 minutes. It is also possible to use PGO to find the critical optimizations done in the PGO optimized build that lead to most of the gains and add annotations in the code (branch taken, branch not taken, force inline, never inline, etc) or split functions the way the PGO optimized build does (e.g. common case is inline the guarding if statement at the beginning of the function, not inline the rest of the function).
pron 7y ago
> What are your thoughts on whole-program optimizing compilers on this? They're awesome, but they're not powerful enough to allow the "cost-free use" as well as JITs. Even whole program optimizing compilers can't be as aggressive as JITs in the general case because, like all AOT compilers, they are limited to what they can prove, and there are many things compilers simply cannot prove (at least not without extreme help from the programmer). Now, don't get me wrong: AOT compilers are often good enough (for languages specially designed for them), but the automatic "zero-cost-use" of JITs is also often good enough compared to cost-free abstractions. We're talking about a general philosophy taken to certain extremes. I would say that a whole-program optimization on a language like ML is a middle ground between those two extremes. > and makes JIT seem like a joke. There's nothing stopping a JIT from doing whole-program optimization. It's just that they can do better for less work. They can do the same optimizations, only with local reasoning because they're not required to prove their correctness. True, they will need to add a guard for some of those speculative optimizations, but on the other hand, they can do so much more (speculate and optimize cases even a whole-program AOT can't prove). Of course, as I said in my original comment, JITs do have other costs.
- pron 7y ago
  P.S. This talk, about a whole-program optimizing compiler for Java bytecode compares the approach to a JIT at ~25-35m: https://youtu.be/CZVa4PBv2qg https://youtu.be/CZVa4PBv2qg This paper discusses the optimizations available to WPO-AOT and JIT: https://dl.acm.org/citation.cfm?doid=3136000.3136002 https://dl.acm.org/citation.cfm?doid=3136000.3136002

denormalfloat 7y ago

Whole program optimization is good if your inputs never change, and you have a infrastructure to benchmark your build and recompile with profiling on each release. JITs do this automatically. A team at work had this release process. 1. Cut the branch at around midnight 2. Run the build with a portion of production traffic for a few hours 3. Collect the profiling info and feed it back into the build. 4. Repush the binary to be tested again. 5. In the morning, the team would manually push the binary to prod. The benefit was clear: about 20% reduction in CPU. However, to get to this level of automation is not easy, and you get it out of the box with JITs. One other thing: It's easy to become dependent on such performance gains. The team that had the process got into a difficult situation where there was a bug in one of the releases, couldn't roll back, and had to cherry pick a change. A few lines of code change had to go through the whole push-profile-rebuild-test cycle before it could be rolled out. Pushing the non-profiled change would have caused several other latency SLO to be violated and pagers fired. Instead, they had to wait the several hours with the bug in place, stressing over how soon the profiling would be done.

wolf550e 7y ago
Better PGO tooling can use the profiles from previous version of the code, which is almost but not quite the same, to compile a PGO optimized build of the patched version. If there is no tooling to do that, a subset of the training data can be used which can be processed in a short amount of time to gather enough profile data to get most of the benefits. So say, instead of 20% faster code after 6 hours, 10% faster code after 15 minutes. It is also possible to use PGO to find the critical optimizations done in the PGO optimized build that lead to most of the gains and add annotations in the code (branch taken, branch not taken, force inline, never inline, etc) or split functions the way the PGO optimized build does (e.g. common case is inline the guarding if statement at the beginning of the function, not inline the rest of the function).

pron 7y ago

> What are your thoughts on whole-program optimizing compilers on this? They're awesome, but they're not powerful enough to allow the "cost-free use" as well as JITs. Even whole program optimizing compilers can't be as aggressive as JITs in the general case because, like all AOT compilers, they are limited to what they can prove, and there are many things compilers simply cannot prove (at least not without extreme help from the programmer). Now, don't get me wrong: AOT compilers are often good enough (for languages specially designed for them), but the automatic "zero-cost-use" of JITs is also often good enough compared to cost-free abstractions. We're talking about a general philosophy taken to certain extremes. I would say that a whole-program optimization on a language like ML is a middle ground between those two extremes. > and makes JIT seem like a joke. There's nothing stopping a JIT from doing whole-program optimization. It's just that they can do better for less work. They can do the same optimizations, only with local reasoning because they're not required to prove their correctness. True, they will need to add a guard for some of those speculative optimizations, but on the other hand, they can do so much more (speculate and optimize cases even a whole-program AOT can't prove). Of course, as I said in my original comment, JITs do have other costs.

pron 7y ago
P.S. This talk, about a whole-program optimizing compiler for Java bytecode compares the approach to a JIT at ~25-35m: https://youtu.be/CZVa4PBv2qg https://youtu.be/CZVa4PBv2qg This paper discusses the optimizations available to WPO-AOT and JIT: https://dl.acm.org/citation.cfm?doid=3136000.3136002 https://dl.acm.org/citation.cfm?doid=3136000.3136002