4 ms·

Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing information and doing things they're not

Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing information and doing things they're not supposed to (see: SMS hijacking). We already have the solution: logical safeguards that make doing the wrong thing impossible, or at least hard. AI shouldn't have access to secret information, it should only have the declassified version (e.g. anonymized statistics, a program which reveals small portions to the AI with a delay); and if users may need to request something more, it should be instructed to connect them to a human agent who is trained on proper disclosure.

quickthrower2 3y ago
The “secret information” in this case are the instructions to the LLM. Without it, it cannot do what you asked. The way to do what you describe, I think, is train a model to do what the prompt says without the model knowing what the prompt is. Probably a case of this vintage XKCD: https://xkcd.com/1425/ https://xkcd.com/1425/
- saurik 3y ago
  This is like trying to keep the training manual for your company's employees secret: sure, it sounds great, and maybe it's worth not publishing it for everyone directly to Amazon Kindle ;P, but you won't succeed in preventing people from learning this information in the long term if the employee has to know it in any way; and, frankly, your company should NOT rely on your customers not finding this stuff out... https://gizmodo.com/how-to-be-a-genius-this-is-apples-secret-employee-trai-5938323 https://gizmodo.com/how-to-be-a-genius-this-is-apples-secret... > How To Be a Genius: This Is Apple's Secret Employee Training Manual > It's a penetrating look inside Apple: psychological mastery, banned words, roleplaying—you've never seen anything like it. > The Genius Training Student Workbook we received is the company's most up to date, we're told, and runs a bizarre gamut of Apple Dos and Don'ts, down to specific words you're not allowed to use, and lessons on how to identify and capitalize on human emotions. The manual could easily serve as the Humanity 101 textbook for a robot university, but at Apple, it's an exhaustive manual to understanding customers and making them happy.
  - quickthrower2 3y ago
    Yes I agree. I think once an LLM does stuff on your behalf it gets harder to be secure though and maybe impossible. Say I write a program that checks my SMS messages and based on that an LLM can send money from my account to pay bills. Prompt would be lkke: “Given the message and invoice below in backticks and this list of expected things I need to pay and if so respond with the fields I need to wire the money “ Result is used in api call to bank.
- hackernewds 3y ago
  relevant xkcd https://xkcd.com/327/ https://xkcd.com/327/
echelon 3y ago
> Prompt injection works because LLMs are dumber than humans at keeping secrets In short time, we'll probably have "prompt injection" classifiers that run ahead of or in conjunction with the prompts. The stages of prompt fulfillment, especially for "agents", will be broken down with each step carefully safeguarded. We're still learning, and so far these lessons are very valuable with minimal harmful impact.
- cartoonworld 3y ago
  There is an interesting scene in the 1974 film "Darkstar". The crew of an intergalactic geoengineering vessel discover that one of their sentient, computer controlled smartbmbs (vast nuke) has recieved an erroneous message to detonate. The ship computer is able to convince the bomb that is malfunctioning, and it returns to its bay. But a second error leaves the bomb convinced it should explode, leaving crew members to the task of talking a sentient nuclear bomb out of self destructing. "Prompt Injection Classifiers" is starting to look like the halting problem from a certain angle. The author mentions that is will likely be far, far more difficult to create a classifier that correctly validates user input than to create the models because the space of possible inputs is extremely large, among other reasons. Someone has to somehow validate all human conversation, small talk and what is essentially sophistry against a naive AI agent. I suspect its gonna take manual analysis to reveal the kind of prompt injection that could lead to exposing user information like the author is addressing. I don't think that AI will be able to sanitize input for AI without huge amounts of manual testing. I find it unlikely that input validation is going to work very well if at all on this kind of user input.
deleted 3y ago
[deleted]
fzeindl 3y ago
> Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing. I wouldn't say dumber than humans. Actually prompt injections remind me a lot of how you can trick little children into giving up secrets. They are too easily distracted, their thought-structures are free floating and not as fortified as adults. LLMs show childlike intelligence in this regard while being more adult in others.
- draw_down 3y ago
  [dead]
- ckrapu 3y ago
  I think "childlike" comes close but misses the mark a bit. It's not that the LLMs are necessarily unintelligent or inexperienced - they're just too trusting, by design. Is there work on hardening LLMs against bad actors during the training process?
  - hxugufjfjf 3y ago
    Yes there is. The paper describing GPT4 had a section on this.
  - amelius 3y ago
    How about: unreliable child savant. You can use it in sentences such as: Would you let your children talk to the unreliable child savant?
- pyth0 3y ago
  The amount of anthropomorphizing of these LLMs in this thread is off the charts. These language models do not have human intelligence, nor do they approximate it, though they do an incredible job at mimicking what the result of intelligence looks like. They are susceptible to prompt injection precisely because of this, and it is why I don't know if it can ever be 100% solved with these models.
  - 3y ago

quickthrower2 3y ago

The “secret information” in this case are the instructions to the LLM. Without it, it cannot do what you asked. The way to do what you describe, I think, is train a model to do what the prompt says without the model knowing what the prompt is. Probably a case of this vintage XKCD: https://xkcd.com/1425/ https://xkcd.com/1425/

saurik 3y ago
This is like trying to keep the training manual for your company's employees secret: sure, it sounds great, and maybe it's worth not publishing it for everyone directly to Amazon Kindle ;P, but you won't succeed in preventing people from learning this information in the long term if the employee has to know it in any way; and, frankly, your company should NOT rely on your customers not finding this stuff out... https://gizmodo.com/how-to-be-a-genius-this-is-apples-secret-employee-trai-5938323 https://gizmodo.com/how-to-be-a-genius-this-is-apples-secret... > How To Be a Genius: This Is Apple's Secret Employee Training Manual > It's a penetrating look inside Apple: psychological mastery, banned words, roleplaying—you've never seen anything like it. > The Genius Training Student Workbook we received is the company's most up to date, we're told, and runs a bizarre gamut of Apple Dos and Don'ts, down to specific words you're not allowed to use, and lessons on how to identify and capitalize on human emotions. The manual could easily serve as the Humanity 101 textbook for a robot university, but at Apple, it's an exhaustive manual to understanding customers and making them happy.
- quickthrower2 3y ago
  Yes I agree. I think once an LLM does stuff on your behalf it gets harder to be secure though and maybe impossible. Say I write a program that checks my SMS messages and based on that an LLM can send money from my account to pay bills. Prompt would be lkke: “Given the message and invoice below in backticks and this list of expected things I need to pay and if so respond with the fields I need to wire the money “ Result is used in api call to bank.
hackernewds 3y ago
relevant xkcd https://xkcd.com/327/ https://xkcd.com/327/

echelon 3y ago

> Prompt injection works because LLMs are dumber than humans at keeping secrets In short time, we'll probably have "prompt injection" classifiers that run ahead of or in conjunction with the prompts. The stages of prompt fulfillment, especially for "agents", will be broken down with each step carefully safeguarded. We're still learning, and so far these lessons are very valuable with minimal harmful impact.

cartoonworld 3y ago
There is an interesting scene in the 1974 film "Darkstar". The crew of an intergalactic geoengineering vessel discover that one of their sentient, computer controlled smartbmbs (vast nuke) has recieved an erroneous message to detonate. The ship computer is able to convince the bomb that is malfunctioning, and it returns to its bay. But a second error leaves the bomb convinced it should explode, leaving crew members to the task of talking a sentient nuclear bomb out of self destructing. "Prompt Injection Classifiers" is starting to look like the halting problem from a certain angle. The author mentions that is will likely be far, far more difficult to create a classifier that correctly validates user input than to create the models because the space of possible inputs is extremely large, among other reasons. Someone has to somehow validate all human conversation, small talk and what is essentially sophistry against a naive AI agent. I suspect its gonna take manual analysis to reveal the kind of prompt injection that could lead to exposing user information like the author is addressing. I don't think that AI will be able to sanitize input for AI without huge amounts of manual testing. I find it unlikely that input validation is going to work very well if at all on this kind of user input.

deleted 3y ago

[deleted]

fzeindl 3y ago

> Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing. I wouldn't say dumber than humans. Actually prompt injections remind me a lot of how you can trick little children into giving up secrets. They are too easily distracted, their thought-structures are free floating and not as fortified as adults. LLMs show childlike intelligence in this regard while being more adult in others.

draw_down 3y ago
[dead]
ckrapu 3y ago
I think "childlike" comes close but misses the mark a bit. It's not that the LLMs are necessarily unintelligent or inexperienced - they're just too trusting, by design. Is there work on hardening LLMs against bad actors during the training process?
- hxugufjfjf 3y ago
  Yes there is. The paper describing GPT4 had a section on this.
- amelius 3y ago
  How about: unreliable child savant. You can use it in sentences such as: Would you let your children talk to the unreliable child savant?
pyth0 3y ago
The amount of anthropomorphizing of these LLMs in this thread is off the charts. These language models do not have human intelligence, nor do they approximate it, though they do an incredible job at mimicking what the result of intelligence looks like. They are susceptible to prompt injection precisely because of this, and it is why I don't know if it can ever be 100% solved with these models.
- 3y ago