3 ms·

Why not just have distinct input streams, or a metadata stream which annotates text in the main stream according to priority in case of conflicting instructions

• simonw 2mo ago
  Because nobody has figured out how to make that work 100% reliably yet. The current approach is to use delimiters that are special tokens that can't be represented in regular text: https://github.com/openai/harmony/blob/main/docs/format.md#special-tokens https://github.com/openai/harmony/blob/main/docs/format.md#s... Then you train your model to take those tokens into account. Which sounds promising... until you see results like this one: https://arxiv.org/abs/2603.12277 https://arxiv.org/abs/2603.12277 > We trace prompt injection to role confusion: models perceive the source of text from how it sounds, not its labeled role. A command hidden in a webpage hijacks an agent simply because it sounds like <user> text, despite its <tool> label