3 ms·Lexing/tokenization implies creating tokens to pass to another parser. This is more general than that.by pubby 6y agoLexing/tokenization implies creating tokens to pass to another parser. This is more general than that.