On regexes, he is generalizing from a small number of examples. Some lexers are straightforward to write by hand, but others are better done with a code genera

On regexes, he is generalizing from a small number of examples. Some lexers are straightforward to write by hand, but others are better done with a code generator. edit: I should really say that you should consider using regular languages to lex your programming languages, not (Perl-style) regexes. Those are different things: https://swtch.com/~rsc/regexp/ https://swtch.com/~rsc/regexp/ I used re2c for Oil and it saves around 5K-10K lines of "groveling through backslashes and braces one a time" (e.g. what other shells do) http://www.oilshell.org/blog/2019/12/22.html#appendix-a-oils-lexer-uses-two-stages-of-code-generation http://www.oilshell.org/blog/2019/12/22.html#appendix-a-oils... And the parser is faster than bash overall: http://www.oilshell.org/blog/2020/01/parser-benchmarks.html http://www.oilshell.org/blog/2020/01/parser-benchmarks.html