I asked myself the same question when I learned yacc and bison. For most languages, the tokens are a regular language and a simple regex lexer is enough. Regul

I asked myself the same question when I learned yacc and bison. For most languages, the tokens are a regular language and a simple regex lexer is enough. Regular expressions used by a lexer end up as a state machine. It does not make much sense to use a full recursive descent parser just to do lexing, as it will be less efficient and use more memory. Consider that the amount of characters ingested by the lexer is far greater than the tokens ingested by the parser.