Note: I like the way you think! You might eliminate a few per-token function calls and cache thrashing by making it less of a 'pull' tokenizer interface. I.e.,

Note: I like the way you think! You might eliminate a few per-token function calls and cache thrashing by making it less of a 'pull' tokenizer interface. I.e., instead of the (elegant) method having Lexer::scan(&t) scan and produce just one token, scan a batch of them at once and fill a contiguous array. Also, your Token class looks like it might be 7 or 8 pointers big. Seems like that could probably slim down a bit.