Lexing translates raw characters into non-semantic tokens. Punctuation is a good way to demonstrate what I mean by non-semantic. Take Rust, which uses `{` for b

Lexing translates raw characters into non-semantic tokens. Punctuation is a good way to demonstrate what I mean by non-semantic. Take Rust, which uses `{` for both blocks and closure literal expressions; and uses `<` for generic parameter/argument wrappers, and also a binary operator. While lexing, you don't try to determine whether `{` is opening a closure or a block, or whether `<` is a binary operator. You just turn them into the tokens `LeftCurlyBrace` and `LeftAngleBracket`. It's the parser's responsibility to take those tokens and their neighbors and figure out what they're being used for. The benefit is that this keeps each stage much simpler: lexing doesn't have to do much if any lookaround, and parsing mostly doesn't have to do anything character-wise.