5 ms·

Prompt injection allows 3rd-party text which the user may not have validated to give LLMs malicious instructions against the wishes of the user. The name "promp

Prompt injection allows 3rd-party text which the user may not have validated to give LLMs malicious instructions against the wishes of the user. The name "prompt injection" often confuses people, but it is a much broader category of attack than jailbreaking or prompt leaking. > the "personal AI assistant" is the best example, since prompt injection means that any time an LLM has access to both private data and untrusted inputs (like emails it has to summarize) there is a risk of something going wrong: https://simonwillison.net/2023/May/2/prompt-injection-explained/#prompt-injection.007 https://simonwillison.net/2023/May/2/prompt-injection-explai... Simon's article here is a really good resource for understanding more about prompt injection (and his other writing on the topic is similarly quite good). I would highly recommend giving it a read, it does a great job of outlining some of the potential risks.

lightbendover 3y ago
The biggest risk to that security risk is its own name. Needs rebranding asap.
- danShumway 3y ago
  :) You're definitely not the first person to suggest that, and there is a decent argument to be made for rebranding. I'm not opposed to it. And I have seen a few at least individual efforts to use different wording, but unfortunately none of them seem to have caught on more broadly (yet), and I'm not sure if there's a clear community consensus yet among security professionals about what they'd prefer to use instead (people who are more embedded in that space than me are welcome to correct me if wrong on that). But I'm at least happy to jump to other terminology if that changes, I do think that calling it "prompt injection" confuses people. I think I remember there being some effort a while back to build a more extensive classification of LLM vulnerabilities that could be used for vulnerability reporting/triaging, but I don't know what the finished project ended up being or what the full details were.
  - jstarfish 3y ago
    Just call it what it is-- social engineering (really, manipulation). "Injection" is a narrow and irrelevant definition. Natural language does not follow a bounded syntax, and injection of words is only one way to "break" the LLM. Buffer overflow works just as well-- smalltalk it to death, until the context outweighs the system prompt. Use lots of innuendo and ambiguous verbiage. After enough discussion of cork soakers and coke sackers you can get LLMs to alliterate about anything. There's nothing injected there, it's just a conversation that went a direction you didn't want to support. In meatspace, if you go to a bank and start up with elaborate stories about your in-laws until the teller forgets what you came in for, or confuse the shit out of her by prefacing everything you say with "today is opposite day," or flash a fake badge and say you're Detective Columbo and everybody needs to evacuate the building, you've successfully managed to get the teller to break protocol. Yet when we do it to LLMs, we give it the woo-woo euphemism "jailbreaking" as though all life descended from iPhones. When the only tool in your box is a computer, every problem is couched in software. It smells like we're trying to redefine manipulation, which does little to help anybody. These same abuses of perception have been employed by and against us for thousands of years already under the names of statecraft, spycraft and stagecraft.
    - simonw 3y ago
      I think you may be confusing jailbreaking and prompt injection. Jailbreaking is more akin to social engineering - it's when you try and convince the model to do something it's "not supposed" to do. Prompt injection is a related but different thing. It's when you take a prompt from a developer - "Translate the following from English to French:" - and then concatenate on a string of untrusted text from a user. That's why it's called "prompt injection" - it's analogous to SQL injection, which was caused by the same mistake, concatenating together trusted instructions with untrusted input.
- ethanbond 3y ago
  Seems directly analogous to SQL injection, no?
  - simonw 3y ago
    Almost. That's why I suggested the name "prompt injection" - because both attacks involve concatenating together trusted and untrusted text. The problem is that SQL injection has an easy fix: you can use parameterized queries, or correctly escape the untrusted content. When I coined "prompt injection" I assumed the fix would look the same. 14 months later it's abundantly clear that implementing an equivalent of those fixes for LLMs is difficult to the point of maybe being impossible, at least against current transformer-based architectures. This means the name "prompt injection" may de-emphasize the scale of the threat!
    - ethanbond 3y ago
      That makes a ton of sense. Well, keen to hear what you (or The People) come up with as a more suitable alternative.
  - scotty79 3y ago
    Same that any scandal is analogous to Watergate (hotel). It makes no sense but since it sounds cool now people will run with it forever.
    - ethanbond 3y ago
      Not really
parineum 3y ago
It should be interpreted similarly as SQL injection. If an LLM has access to private data and is vulnerable to prompt injection, the private data can be compromised.
- danShumway 3y ago
  > as similarly as SQL injection. I really like this analogy, although I would broaden it -- I like to equate it more to XSS: 3rd-party input can change the LLM's behavior, and leaking private data is one of the risks but really any action or permission that the LLM has can be exploited. If an LLM can send an email without external confirmation, than an attacker can send emails on the user's behalf. If it can turn your smart lights on, then a 3rd-party attacker can turn your smart lights on. It's like an attacker being able to run arbitrary code in the context of the LLM's execution environment. My one caveat is that I promised someone a while back that I would always mention when talking about SQL injection that defending against prompt injection is not the same as escaping input to an SQL query or to `innerHTML`. The fundamental nature of why models are vulnerable to prompt injection is very different from XSS or SQL injection and likely can't be fixed using similar strategies. So the underlying mechanics are very different from an SQL injection. But in terms of consequences I do like that analogy -- think of it like a 3rd-party being able to smuggle commands into an environment where they shouldn't have execution privileges.
  - parineum 3y ago
    I totally agree with you. I use the analogy exactly because of the differences in the solution to it that you point out and that, at this point, it seems like an impossible problem to solve. The only solution is to not allow LLMs access to private data. It's definitely a "garden path" analogy meant to lead to that conclusion.

lightbendover 3y ago

The biggest risk to that security risk is its own name. Needs rebranding asap.

danShumway 3y ago
:) You're definitely not the first person to suggest that, and there is a decent argument to be made for rebranding. I'm not opposed to it. And I have seen a few at least individual efforts to use different wording, but unfortunately none of them seem to have caught on more broadly (yet), and I'm not sure if there's a clear community consensus yet among security professionals about what they'd prefer to use instead (people who are more embedded in that space than me are welcome to correct me if wrong on that). But I'm at least happy to jump to other terminology if that changes, I do think that calling it "prompt injection" confuses people. I think I remember there being some effort a while back to build a more extensive classification of LLM vulnerabilities that could be used for vulnerability reporting/triaging, but I don't know what the finished project ended up being or what the full details were.
- jstarfish 3y ago
  Just call it what it is-- social engineering (really, manipulation). "Injection" is a narrow and irrelevant definition. Natural language does not follow a bounded syntax, and injection of words is only one way to "break" the LLM. Buffer overflow works just as well-- smalltalk it to death, until the context outweighs the system prompt. Use lots of innuendo and ambiguous verbiage. After enough discussion of cork soakers and coke sackers you can get LLMs to alliterate about anything. There's nothing injected there, it's just a conversation that went a direction you didn't want to support. In meatspace, if you go to a bank and start up with elaborate stories about your in-laws until the teller forgets what you came in for, or confuse the shit out of her by prefacing everything you say with "today is opposite day," or flash a fake badge and say you're Detective Columbo and everybody needs to evacuate the building, you've successfully managed to get the teller to break protocol. Yet when we do it to LLMs, we give it the woo-woo euphemism "jailbreaking" as though all life descended from iPhones. When the only tool in your box is a computer, every problem is couched in software. It smells like we're trying to redefine manipulation, which does little to help anybody. These same abuses of perception have been employed by and against us for thousands of years already under the names of statecraft, spycraft and stagecraft.
  - simonw 3y ago
    I think you may be confusing jailbreaking and prompt injection. Jailbreaking is more akin to social engineering - it's when you try and convince the model to do something it's "not supposed" to do. Prompt injection is a related but different thing. It's when you take a prompt from a developer - "Translate the following from English to French:" - and then concatenate on a string of untrusted text from a user. That's why it's called "prompt injection" - it's analogous to SQL injection, which was caused by the same mistake, concatenating together trusted instructions with untrusted input.
ethanbond 3y ago
Seems directly analogous to SQL injection, no?
- simonw 3y ago
  Almost. That's why I suggested the name "prompt injection" - because both attacks involve concatenating together trusted and untrusted text. The problem is that SQL injection has an easy fix: you can use parameterized queries, or correctly escape the untrusted content. When I coined "prompt injection" I assumed the fix would look the same. 14 months later it's abundantly clear that implementing an equivalent of those fixes for LLMs is difficult to the point of maybe being impossible, at least against current transformer-based architectures. This means the name "prompt injection" may de-emphasize the scale of the threat!
  - ethanbond 3y ago
    That makes a ton of sense. Well, keen to hear what you (or The People) come up with as a more suitable alternative.
- scotty79 3y ago
  Same that any scandal is analogous to Watergate (hotel). It makes no sense but since it sounds cool now people will run with it forever.
  - ethanbond 3y ago
    Not really

parineum 3y ago

It should be interpreted similarly as SQL injection. If an LLM has access to private data and is vulnerable to prompt injection, the private data can be compromised.

danShumway 3y ago
> as similarly as SQL injection. I really like this analogy, although I would broaden it -- I like to equate it more to XSS: 3rd-party input can change the LLM's behavior, and leaking private data is one of the risks but really any action or permission that the LLM has can be exploited. If an LLM can send an email without external confirmation, than an attacker can send emails on the user's behalf. If it can turn your smart lights on, then a 3rd-party attacker can turn your smart lights on. It's like an attacker being able to run arbitrary code in the context of the LLM's execution environment. My one caveat is that I promised someone a while back that I would always mention when talking about SQL injection that defending against prompt injection is not the same as escaping input to an SQL query or to `innerHTML`. The fundamental nature of why models are vulnerable to prompt injection is very different from XSS or SQL injection and likely can't be fixed using similar strategies. So the underlying mechanics are very different from an SQL injection. But in terms of consequences I do like that analogy -- think of it like a 3rd-party being able to smuggle commands into an environment where they shouldn't have execution privileges.
- parineum 3y ago
  I totally agree with you. I use the analogy exactly because of the differences in the solution to it that you point out and that, at this point, it seems like an impossible problem to solve. The only solution is to not allow LLMs access to private data. It's definitely a "garden path" analogy meant to lead to that conclusion.