3 ms·

Correct me if I am wrong, but the way I understand is that, when LLMs have to process a certain text, every word will get tokenized into some vector representat

Correct me if I am wrong, but the way I understand is that, when LLMs have to process a certain text, every word will get tokenized into some vector representation. So, if you insert the new special token and wrap data around, it is not the fact that you can ignore the entire prompt. Because as soon as you have to prompt the model, you will be using the entire tokenized sentence. This would mean that even if there is a special token somewhere, the model will not be able to ignore the token before/after that special token. So what will happen to the model if somewhere there is prompt that overrides this special token?

sillysaurusx 4y ago
> So what will happen to the model if somewhere there is prompt that overrides this special token? The model will be trained so that data within those special token pairs can't override the prompt, similar to how strings in an SQL query can't override the query: it's escaped. As for "how," it's a matter of using RLHF to punish the model for failing to do this. The reason I'm optimistic this is a solid answer is because attackers can't insert those special tokens. They're meta-tokens, which only OpenAI/Microsoft have access to. So you can't break out of the sandbox that it was trained to ignore.