4 ms·
Free speedups are always appreciated but when benchmarking some not-optimized-at-all lexers, they were still so fast that I couldn't really justify optimizing t
by mhh__ 5y ago
Free speedups are always appreciated but when benchmarking some not-optimized-at-all lexers, they were still so fast that I couldn't really justify optimizing them.
Unless you're parsing genuinely enormous amounts of code (or latency bound) I find that most compilers are far too slow in other areas to justify optimizing the lexer versus fixing architectural issues in (say) semantic analysis.
A lot of parts of compilers are non-local, and involve lots of pointer chasing, which means avoiding repeated work is very important. In the D compiler there a bunch of checks where we stop you shooting yourself in the foot, I think generating all of these tree-walkers from a specification (so you can walk it all in one go) would probably be a decent speedup.
- woodruffw 5y agoIn this case, the speedup would probably be during LLVM's build itself: that's when TableGen inputs are parsed as part of codegen for big blobs of C++. Improvements to lexing speed here could actually produce some build time improvements, since TableGen's outputs tend to have a high dependent degree in the dependency graph. (That being said, I have no idea if LLVM would merge a TableGen lexer written in Go.)
- mhh__ 5y agoThe question prior to any of that is how many lines of tablegen LLVM has According to a cloc definition I just did: LLVM Table Gen files: 1041 blank: 75684 comment: 69272 code: 480047 Probably enough to warrant a faster lexer.
- klodolph 5y agoI’ve seen at least a couple cases where lexer time dominated overall compile time.
- rob74 5y agoOne part of Go that really needs optimizing is the regexp library. It's still a pretty "basic" implementation and can't keep up with the optimized implementations of other languages...