3 ms·
Another alternative would be the use of a string interner, with the tokens storing the interner ID. Advantages would be that the token type can stay small and
by Measter 4y ago
Another alternative would be the use of a string interner, with the tokens storing the interner ID.
Advantages would be that the token type can stay small and Copy, while not having a lifetime to carry around.
Disadvantages would be the overhead of the interning, which would slow down lexing, and you'd need to drag around the interner to anywhere you need the actual string.
- vlmutolo 4y agoI wonder if string interning is advantageous for a lexer, where you know the strings you'd want to intern ahead-of-time (AoT). If you have reserved words, those will probably end up as enum variants in "Token". And things you can't know AoT are less likely to be amenable to internment, like comments and string literals. Tokenization already includes something like a manual string internment process. Originally I had the same thought as you, and that's why I hunted down the smol_str library. I knew that Aleksey used it in his rust-analyzer parser and figured it was an interner. But then I saw it wasn't (other than for whitespace kinda) and started to wonder string interning fit this problem.