4 ms·

Shouldn’t the user prompt be a separate input channel from emails etc?

by geon 3y ago

danShumway 3y ago
Case in point for my reply to Simon on why I draw this distinction :) The natural instinct for many people about prompt injection is to assume that contextual separation between data and commands is enough to solve the problem. The short answer is: A) we don't know how to enforce that separation with current LLMs. B) Even if we did, there is another category of attack (or Simon might argue if I understand his position, a sub-category of prompt injection) that can not be defended against by separating untrusted input and system/user prompts.
simonw 3y ago
Yes, and it sort-of is - you can put the instructions in the system prompt and the untrusted data in the regular prompt. The problem is... that doesn't work! Modern LLMs are trained to put more "weight" on instructions in the system prompt, but it's still all a stream of tokens in the end, and I've never seen a system prompt that can't be over-ridden by the non-system prompt given enough tokens (like a multi-paragraph email).
- geon 3y ago
  But the system prompt is just what comes first on the same channel. Having entirely separate inputs can't be a novel idea, can it? It might be enough to have a set of input neurons to indicate what channel the current input belongs to. After that, I imagine the nn could be trained simon-says-style with commands in the email channel, and punishment if it responds at all to them.
  - danShumway 3y ago
    It's been a while since I dug into this, but my understanding is, two issues: 1. Reinforcement training is just that: reinforcement. It doesn't completely eliminate behaviors. My understanding is that OpenAI is already trying to do some reinforcement training to block some prompt injections, but you can't get past the problem that fundamentally the network is still trained first on a massive amount of data that does not follow that rule. Think of it this way, if reinforcement training isn't enough to get a model to "forget" how to swear, it's also not going to be enough to get a model to do something more complicated (distinguish channels for instructions). And while it's been a little while since I was paying attention to the LLM space, the last time I looked into it there was a growing consensus among the researchers I was following that reinforcement training could never completely eliminate behaviors. 2. It's worth noting that LLMs aren't initially trained to have a concept of instructions at all. That LLMs can obey instructions is an emergent property from their training on text prediction. So we're sort of pushing against the grain in trying to add fundamental basic properties to how LLMs understand and follow instructions because we don't 100% know what is going on when an LLM follows instructions. This sort of gets at what Simon is talking about with everything just being a stream of tokens. LLMs continue streams of tokens, and as a consequence of doing that, they also end up following instructions. There aren't multiple primitives to work with, all of the primitives are streams of tokens, and the best we can do with current training methods is after the fact try to get the LLM to treat those tokens differently. But at its core, the LLM is still going to complete streams of tokens, there isn't (using current training methods) any notion of contextual input. And it's possible to think about a theoretical LLM that was trained using contextual input, but in reality -- where would you get the training data to do that? Current LLMs are trained on massive amounts of data, basically the entire Internet. So any theories about how to change that training process need to be compatible with the training data that actually exists, and the training data that exists doesn't have a concept of context separation. You can add that afterwards using reinforcement training to some degree and AI companies are trying to do this, but that's very complicated (Simon says training is probably not going to be sufficient to guard against the majority of prompt injections). Which... see point #1. ---- I guess add issue #3, to reiterate; contextual separation isn't enough to completely secure LLMs because context is necessarily fuzzy for any agent that interprets data. For LLMs to be useful as general tools, they need that fuzziness and they need to be suggestable and they need to follow instructions. But all of that extra capability beyond an agent's assigned task is vulnerability. And so a general-purpose AI for multiple tasks that is also immune to prompt injection and can be used in security-critical applications might be a logical contradiction? I'm not sure that thing exists or could exist.

Shouldn’t the user prompt be a separate input channel from emails etc?

by geon 3y ago

Shouldn’t the user prompt be a separate input channel from emails etc?

Case in point for my reply to Simon on why I draw this distinction :) The natural instinct for many people about prompt injection is to assume that contextual separation between data and commands is enough to solve the problem. The short answer is: A) we don't know how to enforce that separation with current LLMs. B) Even if we did, there is another category of attack (or Simon might argue if I understand his position, a sub-category of prompt injection) that can not be defended against by separating untrusted input and system/user prompts.

simonw 3y ago

Yes, and it sort-of is - you can put the instructions in the system prompt and the untrusted data in the regular prompt. The problem is... that doesn't work! Modern LLMs are trained to put more "weight" on instructions in the system prompt, but it's still all a stream of tokens in the end, and I've never seen a system prompt that can't be over-ridden by the non-system prompt given enough tokens (like a multi-paragraph email).

geon 3y ago
But the system prompt is just what comes first on the same channel. Having entirely separate inputs can't be a novel idea, can it? It might be enough to have a set of input neurons to indicate what channel the current input belongs to. After that, I imagine the nn could be trained simon-says-style with commands in the email channel, and punishment if it responds at all to them.
- danShumway 3y ago
  It's been a while since I dug into this, but my understanding is, two issues: 1. Reinforcement training is just that: reinforcement. It doesn't completely eliminate behaviors. My understanding is that OpenAI is already trying to do some reinforcement training to block some prompt injections, but you can't get past the problem that fundamentally the network is still trained first on a massive amount of data that does not follow that rule. Think of it this way, if reinforcement training isn't enough to get a model to "forget" how to swear, it's also not going to be enough to get a model to do something more complicated (distinguish channels for instructions). And while it's been a little while since I was paying attention to the LLM space, the last time I looked into it there was a growing consensus among the researchers I was following that reinforcement training could never completely eliminate behaviors. 2. It's worth noting that LLMs aren't initially trained to have a concept of instructions at all. That LLMs can obey instructions is an emergent property from their training on text prediction. So we're sort of pushing against the grain in trying to add fundamental basic properties to how LLMs understand and follow instructions because we don't 100% know what is going on when an LLM follows instructions. This sort of gets at what Simon is talking about with everything just being a stream of tokens. LLMs continue streams of tokens, and as a consequence of doing that, they also end up following instructions. There aren't multiple primitives to work with, all of the primitives are streams of tokens, and the best we can do with current training methods is after the fact try to get the LLM to treat those tokens differently. But at its core, the LLM is still going to complete streams of tokens, there isn't (using current training methods) any notion of contextual input. And it's possible to think about a theoretical LLM that was trained using contextual input, but in reality -- where would you get the training data to do that? Current LLMs are trained on massive amounts of data, basically the entire Internet. So any theories about how to change that training process need to be compatible with the training data that actually exists, and the training data that exists doesn't have a concept of context separation. You can add that afterwards using reinforcement training to some degree and AI companies are trying to do this, but that's very complicated (Simon says training is probably not going to be sufficient to guard against the majority of prompt injections). Which... see point #1. ---- I guess add issue #3, to reiterate; contextual separation isn't enough to completely secure LLMs because context is necessarily fuzzy for any agent that interprets data. For LLMs to be useful as general tools, they need that fuzziness and they need to be suggestable and they need to follow instructions. But all of that extra capability beyond an agent's assigned task is vulnerability. And so a general-purpose AI for multiple tasks that is also immune to prompt injection and can be used in security-critical applications might be a logical contradiction? I'm not sure that thing exists or could exist.