3 ms·
How do the "special tokens" work? Is this a completely reliable mechanism for delimiting the different parts of the prompt? Are they guaranteed to be distinct
by ptx 2y ago
How do the "special tokens" work? Is this a completely reliable mechanism for delimiting the different parts of the prompt?
Are they guaranteed to be distinct from anything that could occur in the prompt, something like JavaScript's Symbol?
Or are they strings that are pretty likely not to occur in the prompt, something like a MIME boundary?
Or are they literally the strings "<|start|>" etc. used to denote them in the spec?
- sharkjacobs 2y agothey are "literally the strings" but I believe they will be escaped, or encoded differently, if a user tries to inject them as part of a prompt.
- jffry 2y agoYeah the tokens are more akin to JS Symbol. If you're parsing untrusted user inputs into tokens, you can make sure your tokenizer will never produce the actual numbers corresponding to those tokens. A simplified example: I can `.charCodeAt` a string all I want but I'll never get a negative number, so I can safely use -1 to mean something special in the transformed sequence of "tokens".