3 ms·

These books are kind of old. Are they still state of the art or is there something newer? I had a compiler class at university but that was a while ago, so I w

These books are kind of old. Are they still state of the art or is there something newer? I had a compiler class at university but that was a while ago, so I would like to update my knowledge because there seems to be a lot of exiting stuff going on with compilers and transpilers.

rwmj 7y ago
The basics don't change. There was this rather fun talk on HN 2 weeks ago https://news.ycombinator.com/item?id=19657983 https://news.ycombinator.com/item?id=19657983 where he asserts that only 8 optimizations are necessary to get about 80% of best case performance, and all of those were catalogued in 1971.
- felixangell 7y ago
  Wow, that's a really interesting fact. I hope you don't mind but I updated the article to reference this and credited you accordingly :^)
eatonphil 7y ago
While some projects within the compiler toolchain reinvent the wheel to varying degrees of increased performance/functionality (e.g. MIR, probably some parsing tools too, gold linker, lldb, etc.), the majority of projects these days take advantage of existing tooling to applying frontends and backends in different ways. Want a JavaScript to LLVM compiler? Map the JavaScript AST using an existing JavaScript parser to the LLVM AST and have LLVM generate code. The combinations are infinite and high quality. Most people (myself included) only experiment at the highest level. All this is to say the barrier to entry in PL/compiler/interpreter implementation is only as high as you want it to be. You can do the whole thing yourself as a learning experience or you can use existing tools to finish a fairly-fully-featured project in a few hours over a few weeks.
jcranmer 7y ago
As others have pointed out, a lot of compilers have basically stayed the same in the past 25 years. What's changed has been some new (generally special-purpose) minor optimizations, and the growth of compilers to consider less traditional domains. The newer things: * SLP vectorization. Instead of classic vectorization (where you have a loop that you convert to a vector), SLP vectorization tries to form vectors from a single basic block, and it can work quite well for the small SIMD units such as SSE. * Polyhedral loop transformation. This is sort of the equivalent of the lattice-based dataflow analysis methodology, which is to say it's a very powerful, and slow, general-purpose technique which actually isn't used all that much in production compilers. * Decompilation, disassembly, and other forms of static binary analysis have progressed a fair amount in the past decade or so. * Dynamic binary analysis/translation, of which the state of the art is probably Intel Pin (https://dl.acm.org/citation.cfm?id=1065034 https://dl.acm.org/citation.cfm?id=1065034). * JITs have evolved a lot. In terms of what's missing from the classic compiler books, this is the big missing area. Things such as tracing, recompilation techniques, garbage collection, guard techniques, etc. have come a long way, and I'm not off-hand aware of any actual good book or paper here that covers most of the topic. * Superoptimization is a technique that's still mostly in the academic phase, but you are seeing peephole optimizations and other information populated by ahead-of-time superoptimization (e.g., Regehr's Souper work). * Symbolic and concolic execution is something else that's in the transition phase between academic and industry work. The most advanced use of concolic execution is fuzzing work where concolic execution is used to guide the fuzzer to generate specific inputs to test as many paths as possible. Most of these topics wouldn't be covered in something as introductory as the Dragon Book (which itself has poor coverage of the major loop transformations), and generally only come into play in more specific scenarios.

rwmj 7y ago

The basics don't change. There was this rather fun talk on HN 2 weeks ago https://news.ycombinator.com/item?id=19657983 https://news.ycombinator.com/item?id=19657983 where he asserts that only 8 optimizations are necessary to get about 80% of best case performance, and all of those were catalogued in 1971.

felixangell 7y ago
Wow, that's a really interesting fact. I hope you don't mind but I updated the article to reference this and credited you accordingly :^)

eatonphil 7y ago

While some projects within the compiler toolchain reinvent the wheel to varying degrees of increased performance/functionality (e.g. MIR, probably some parsing tools too, gold linker, lldb, etc.), the majority of projects these days take advantage of existing tooling to applying frontends and backends in different ways. Want a JavaScript to LLVM compiler? Map the JavaScript AST using an existing JavaScript parser to the LLVM AST and have LLVM generate code. The combinations are infinite and high quality. Most people (myself included) only experiment at the highest level. All this is to say the barrier to entry in PL/compiler/interpreter implementation is only as high as you want it to be. You can do the whole thing yourself as a learning experience or you can use existing tools to finish a fairly-fully-featured project in a few hours over a few weeks.

jcranmer 7y ago

As others have pointed out, a lot of compilers have basically stayed the same in the past 25 years. What's changed has been some new (generally special-purpose) minor optimizations, and the growth of compilers to consider less traditional domains. The newer things: * SLP vectorization. Instead of classic vectorization (where you have a loop that you convert to a vector), SLP vectorization tries to form vectors from a single basic block, and it can work quite well for the small SIMD units such as SSE. * Polyhedral loop transformation. This is sort of the equivalent of the lattice-based dataflow analysis methodology, which is to say it's a very powerful, and slow, general-purpose technique which actually isn't used all that much in production compilers. * Decompilation, disassembly, and other forms of static binary analysis have progressed a fair amount in the past decade or so. * Dynamic binary analysis/translation, of which the state of the art is probably Intel Pin (https://dl.acm.org/citation.cfm?id=1065034 https://dl.acm.org/citation.cfm?id=1065034). * JITs have evolved a lot. In terms of what's missing from the classic compiler books, this is the big missing area. Things such as tracing, recompilation techniques, garbage collection, guard techniques, etc. have come a long way, and I'm not off-hand aware of any actual good book or paper here that covers most of the topic. * Superoptimization is a technique that's still mostly in the academic phase, but you are seeing peephole optimizations and other information populated by ahead-of-time superoptimization (e.g., Regehr's Souper work). * Symbolic and concolic execution is something else that's in the transition phase between academic and industry work. The most advanced use of concolic execution is fuzzing work where concolic execution is used to guide the fuzzer to generate specific inputs to test as many paths as possible. Most of these topics wouldn't be covered in something as introductory as the Dragon Book (which itself has poor coverage of the major loop transformations), and generally only come into play in more specific scenarios.