3 ms·
Yeah the tokens are more akin to JS Symbol. If you're parsing untrusted user inputs into tokens, you can make sure your tokenizer will never produce the actual
by jffry 2y ago
Yeah the tokens are more akin to JS Symbol.
If you're parsing untrusted user inputs into tokens, you can make sure your tokenizer will never produce the actual numbers corresponding to those tokens.
A simplified example: I can `.charCodeAt` a string all I want but I'll never get a negative number, so I can safely use -1 to mean something special in the transformed sequence of "tokens".