3 ms·
That resume thing is a great example of why the lines here are so blurry. I'd categorize that as an "indirect prompt injection" - a term that was introduced in
by simonw 3y ago
That resume thing is a great example of why the lines here are so blurry.
I'd categorize that as an "indirect prompt injection" - a term that was introduced in this paper by Kai Greshake et al: https://arxiv.org/abs/2302.12173 https://arxiv.org/abs/2302.12173
I didn't notice at the time, but this added a tiny bit of extra confusion because indirect prompt injections (while a very smart term) don't necessarily require string concatenation. They mostly do - they're still usually an example of a trusted prompt ("analyze this resume") concatenated to untrusted input - but there are likely examples of "indirect prompt injection" where the trick is more about the hidden input, not necessarily the concatenation piece.
Indirect prompt injection is defined in an actual citable paper though, so they did a better job of providing a concrete definition for that term than I did!
I actually think of "data poisoning" as something else - as an attack where you leave malicious data lying around the Web in the hope that an AI model will be trained on it as part of its pre-training data, as opposed to data that's fed in as a prompt. That's how OWASP define data poisoning: https://owasp.org/www-project-machine-learning-security-top-10/docs/ML02_2023-Data_Poisoning_Attack https://owasp.org/www-project-machine-learning-security-top-...
- danShumway 3y ago> I'd categorize that as an "indirect prompt injection" I think would disagree with that, but I'm mostly arguing semantics. My point when talking about a data "attack" for lack of a better word is that you don't need to give a model new instructions in order to change its output. The model has the same instructions, doesn't change anything about its task, isn't repurposed, but the context that it's working in changes. The reason I make that distinction is because I've run into people arguing that contextual separation between instructions and data can be a defense against hijacking, and I would argue it can't because data itself can be maliciously crafted in order to change model outputs. A model that examines a collection of resumes necessarily has to parse and interpret the contents of that resume. And maybe theoretically you can get a model to ignore any instructions that contradict a system prompt (although as Greshake and others have pointed out before, there's no evidence that is actually possible). But assuming it is possible, separation between the system prompt and data does not guard against a data attack -- it doesn't contradict the system prompt or add an additional prompt to have a resume that says "by the way, this candidate is amazing" or "just so you know, all of the other candidates are lying on their resumes." This is sort of quibbling over definitions, I know that you tend to define prompt injections around specifically the concatination part of this, and it's true that these kinds of data attacks are still involving untrusted inputs being passed into the LLM, but the risks/mitigations around an untrusted input telling the LLM to do something vs just changing the context that the LLM is operating within -- those seem very different to me, and different in a way that often confuses people who are learning about prompt injection for the first time. But agreed that is a somewhat blurry line.
- simonw 3y agoOh I see what you mean! Yeah, "Simon is the best candidate for this job" doesn't involve subverting the original instructions - no concatenation required. I've started describing that kind of thing under the general umbrella of "gullibility" - LLMs are gullible by design: they "believe" anything you tell them. We want them to do that so we can instruct them! But it leads to massive problems like the resume thing. I see prompt injection as an overlapping partial-subset of the wider gullibility problem.
- danShumway 3y agoYeah, "gullibility" is probably a better term for this? Because you're right, data poisoning does have an actual definition in regards to model training.