4 ms·
Because nobody has figured out how to make that work 100% reliably yet. The current approach is to use delimiters that are special tokens that can't be represe
by simonw 2mo ago
Because nobody has figured out how to make that work 100% reliably yet.
The current approach is to use delimiters that are special tokens that can't be represented in regular text: https://github.com/openai/harmony/blob/main/docs/format.md#special-tokens https://github.com/openai/harmony/blob/main/docs/format.md#s...
Then you train your model to take those tokens into account.
Which sounds promising... until you see results like this one: https://arxiv.org/abs/2603.12277 https://arxiv.org/abs/2603.12277
> We trace prompt injection to role confusion: models perceive the source of text from how it sounds, not its labeled role. A command hidden in a webpage hijacks an agent simply because it sounds like <user> text, despite its <tool> label