One benefit of splitting the lexer and parser is that you can write tests for the lexer that cover different lexical variations of a token -- e.g. for decimal/d

One benefit of splitting the lexer and parser is that you can write tests for the lexer that cover different lexical variations of a token -- e.g. for decimal/double values you could test that `1.`, `.2`, `1.0` all produce the same token type. Then, when writing tests for your parser in the parts of the grammar that have that token type you only need one test to cover all the different lexical variations. This way, your parser tests are then about covering the different symbol transitions in the grammar, making them much more manageable. It is possible to have more complex lexers -- typically state-based lexers -- that switch state on given input. You can do this when tokenizing comments, strings, or other constructs so the lexer can emit the correct symbol for things like string interpolation. This can be useful when syntax highlighting or other token-based tasks are separate to parsing. I can see how a state-based lexer and a parser can be combined to avoid managing the state in the lexer -- that is because the parser will know what the current state is due to where it is in the grammar, so can call the appropriate method on the lexer, or pass the corresponding state information. I've not yet experimented with this approach, but it could be useful in some contexts. Note: I've written a lexer and parser for a language plugin for IntelliJ which has separate lexer and parser steps, and uses the lexer for syntax highlighting. I'm investigating support for the language server protocol (LSP), so may also experiment with a directed/orchestrated lexer as described above.