I don't agree. The second version is the typical way hand-written compiler lexers are written, and they're straightforward to add new "states" to, or to modify

I don't agree. The second version is the typical way hand-written compiler lexers are written, and they're straightforward to add new "states" to, or to modify the states. The program counter is where you're "at", and the state space is normally straightforward. In fact it only gets complicated when additional non-control-flow state has been added. A typical hand-written lexer looks a bit like this (EOF testing omitted for clarity): // skip whitespace while isWhiteSpace(peekChar()) nextChar() // determine token type switch (peekChar()) case isAlpha start := currentPosition() nextChar() while isAlphaNumeric(peekChar()) nextChar() tokenType = IDENT tokenValue = substring(start, currentPosition()) case isNumeric start := currentPosition() nextChar() while isNumeric(peekChar()) nextChar() tokenType = INT tokenValue = substring(start, currentPosition()) tokenAsInt = int(tokenValue) case '+' nextChar() tokenType = PLUS // and so on Adding new cases to this is trivial. Nested context-specific lexical details (like escape codes in strings) aren't merged with a global state machine, and can be handed off to subroutines, enabling a modularity which is less confusing than having multiple state machines.