3 ms·
I suspect that lexer generators mostly exist because that's what people who want to implement compilers are being taught. But there's nothing difficult about wr
by dist1ll 3y ago
I suspect that lexer generators mostly exist because that's what people who want to implement compilers are being taught. But there's nothing difficult about writing a lexer. It can be done in less than 100 LoC, easily half that if your language has good string ergonomics and support for pattern matching.
- remexre 3y agoWouldn't the hand-written one be more work to write and measurably slower than the re2c-generated one?
- dist1ll 3y agoIt's really not much work, and it removes another source of dependency/build bloat. Also, if you care about speed, hand-rolling your lexer will always be the fastest (and even naive implementations can rival/outcompete generators).
- ceronman 3y agoNot only that, but a manually written lexer has much more flexibility. It allows you to easily implement things that lexer generators struggle with. Think of significant indentation, nested comments, interpolation inside string literals. Sometimes you can't do this at all with generators, sometimes you can, but code becomes a mess. That's the reason why many popular programming language implementations use a hand written lexer and parser.
- fanf2 3y agore2c makes it easy to drop into hand-written code where necessary. For example, in C backslash-newline makes a huge mess of everything unless you can handle it underneath the lexer, which works nicely in re2c. In C++, raw string literals are not even context-free, let alone regular, so they have to be parsed with hand-written code. This is straightforward with re2c.
- o11c 3y agoThe problem with hand-written lexers/parsers (and some generated parsers, notably PEG these days) is that you have no idea if you can actually trust it. There are so many weird edge cases for lexing and parsing that I would never dare to do it without machine verification.