I wonder if string interning is advantageous for a lexer, where you know the strings you'd want to intern ahead-of-time (AoT). If you have reserved words, those

I wonder if string interning is advantageous for a lexer, where you know the strings you'd want to intern ahead-of-time (AoT). If you have reserved words, those will probably end up as enum variants in "Token". And things you can't know AoT are less likely to be amenable to internment, like comments and string literals. Tokenization already includes something like a manual string internment process. Originally I had the same thought as you, and that's why I hunted down the smol_str library. I knew that Aleksey used it in his rust-analyzer parser and figured it was an interner. But then I saw it wasn't (other than for whitespace kinda) and started to wonder string interning fit this problem.