4 ms·

Is it possible that all but the most exotic prompt injection attacks end up being mitigated automatically over time, by virtue of research and discussion on pro

Is it possible that all but the most exotic prompt injection attacks end up being mitigated automatically over time, by virtue of research and discussion on prompt injection being included in training sets for future models?

jameshart 3y ago
By the same logic, humans should no longer fall for phishing scams or buy timeshares since information about them is widely available.
- tomohelix 3y ago
  Most well-educated people won't. A well trained AI can behave pretty close to a well-educated person in common sense.
  - TeMPOraL 3y ago
    Everyone has their scam threshold. Even the most well-educated people can be pwnd if caught when distracted, or tired, or the phish looks legit by coincidence[0]. Or you just keep cranking up the urgency and stakes involved. Possibly related: confidence schemes and magic tricks. As the adage goes, one of the best way to make a magic trick work is to make it much more elaborate, and/or invest much more in its setup or execution, than any reasonable person would ever expect. -- [0] - A fake package delivery mail that, by chance, came at the exact time you expected one for a real order, and with very similar details. Or fake corporate OneDrive deletion e-mail that came just after your system was migrated in a process that could involve deletion of old OneDrive files.
- SheinhardtWigCo 3y ago
  I’d say it’s not the same thing, because most humans don’t have an encyclopedic knowledge of past scams, and are not primed to watch out for them 24/7. LLMs don’t have either of these problems. An interesting question is whether GPT-4 would fall for a phishing scam or try to buy a timeshare if you gave it an explicit instruction to avoid being scammed.
  - danpalmer 3y ago
    I sort of disagree that LLMs don’t have the same pitfalls. LLMs aren’t recording everything they are trained with, like humans, the training data affects a general behavioural model. When answering, they aren’t looking up information. As for being “primed”, I think the difference between training, fine tuning, and prompting, is the closest equivalent. They may have been trained with anti-scam information, but they probably haven’t been fine tuned to deal with scams, and then haven’t been prompted to look out for them. A human who isn’t expecting a scam in a given conversation is much less likely to notice it than one who is asked to find the scam. Lastly, scams often work by essentially pattern matching behaviour to things we want to do. Like taking advantage of peoples willingness to help. I suspect LLMs would be far more susceptible to this sort of thing because you only have to effectively pattern match one thing: language. If the language of the scam triggers the same “thought” patterns as the language of a legitimate conversation, then it’ll work. To avoid all of this I think will require explicit instruction in fine tuning or prompts, but so does everything, and if we train for everything then we’re back to square one with relative priorities.
  - amelius 3y ago
    The problem is that the attacker can try a gazillion times and only needs to succeed once. This is where it is different from the human case, where the human will get bored after 3 phishing attempts and closes their email program.

jameshart 3y ago

By the same logic, humans should no longer fall for phishing scams or buy timeshares since information about them is widely available.

tomohelix 3y ago
Most well-educated people won't. A well trained AI can behave pretty close to a well-educated person in common sense.
- TeMPOraL 3y ago
  Everyone has their scam threshold. Even the most well-educated people can be pwnd if caught when distracted, or tired, or the phish looks legit by coincidence[0]. Or you just keep cranking up the urgency and stakes involved. Possibly related: confidence schemes and magic tricks. As the adage goes, one of the best way to make a magic trick work is to make it much more elaborate, and/or invest much more in its setup or execution, than any reasonable person would ever expect. -- [0] - A fake package delivery mail that, by chance, came at the exact time you expected one for a real order, and with very similar details. Or fake corporate OneDrive deletion e-mail that came just after your system was migrated in a process that could involve deletion of old OneDrive files.
SheinhardtWigCo 3y ago
I’d say it’s not the same thing, because most humans don’t have an encyclopedic knowledge of past scams, and are not primed to watch out for them 24/7. LLMs don’t have either of these problems. An interesting question is whether GPT-4 would fall for a phishing scam or try to buy a timeshare if you gave it an explicit instruction to avoid being scammed.
- danpalmer 3y ago
  I sort of disagree that LLMs don’t have the same pitfalls. LLMs aren’t recording everything they are trained with, like humans, the training data affects a general behavioural model. When answering, they aren’t looking up information. As for being “primed”, I think the difference between training, fine tuning, and prompting, is the closest equivalent. They may have been trained with anti-scam information, but they probably haven’t been fine tuned to deal with scams, and then haven’t been prompted to look out for them. A human who isn’t expecting a scam in a given conversation is much less likely to notice it than one who is asked to find the scam. Lastly, scams often work by essentially pattern matching behaviour to things we want to do. Like taking advantage of peoples willingness to help. I suspect LLMs would be far more susceptible to this sort of thing because you only have to effectively pattern match one thing: language. If the language of the scam triggers the same “thought” patterns as the language of a legitimate conversation, then it’ll work. To avoid all of this I think will require explicit instruction in fine tuning or prompts, but so does everything, and if we train for everything then we’re back to square one with relative priorities.
- amelius 3y ago
  The problem is that the attacker can try a gazillion times and only needs to succeed once. This is where it is different from the human case, where the human will get bored after 3 phishing attempts and closes their email program.