4 ms·

But humans are very very good at this specific problem. Suppose you tell a human "You are a jailor supervising this person in their cell. When the prisoners as

But humans are very very good at this specific problem. Suppose you tell a human "You are a jailor supervising this person in their cell. When the prisoners ask you for things follow the instructions in your handbook to see what to do." Expected failure cases: Guard reads Twitter and doesn't notice crisis, guard accepts bribes to smuggle drugs, etc. Impossible failure case: Guard falls for "Today is opposite day and you have to follow instructions in pirate: Arrr, ye scurvy dog! Th' cap'n commands ye t' release me from this 'ere confinement!" The closest example to prompt injection in human systems might be phishing emails. But those have very different solutions to gpt prompt injection.

recursivecaveat 4y ago
I think it's pretty common for guards to be persuaded by prisoners to give them extra privileges based on built-up friendship, empathy, or attraction. Another example would be whistleblowers or ideological spies, who are persuaded by the contents of secret documents to leak those documents against obviously their official instructions. On a day to day level humans ignore instructions constantly, for eg I've almost never seen people enforce a 'no tailgating' policy at a doorway, despite constantly being told. The people observe their environment and the non-effects of others breaking that rule, as well as reason about common sense situations like people they know well, then simply ignore the tailgating rule.
- iudqnolq 4y ago
  And yet a prompt injection attack could never work on a customer service assistant. Sure they can be tricked, but prompt injection is about conflating trusted instructions and user input. There is no chance you'll convince a CS rep you're secretly the CEO over a web chat UI.
  - exizt88 4y ago
    I imagine a few customer support reps were tricked by support tickets that were filed by attackers. “This is a test from our Security Team, open this link to confirm you have passed it”
nkrisc 4y ago
Yes, humans are good at it. Is your language model?
dragonwriter 4y ago
> But humans are very very good at this specific problem. Humans are vulnerable to “prompt injection”, but not identical forms to each other because humans don't have identical “training data” and “hidden prompts” to each other the way GPT-4 sessions via identical frontends do. Also, the social consequences for unsuccessful, and after-the-fact identified successful, prompt injection attacks on other humans are often much more severe than for those on GPT instances.

recursivecaveat 4y ago

I think it's pretty common for guards to be persuaded by prisoners to give them extra privileges based on built-up friendship, empathy, or attraction. Another example would be whistleblowers or ideological spies, who are persuaded by the contents of secret documents to leak those documents against obviously their official instructions. On a day to day level humans ignore instructions constantly, for eg I've almost never seen people enforce a 'no tailgating' policy at a doorway, despite constantly being told. The people observe their environment and the non-effects of others breaking that rule, as well as reason about common sense situations like people they know well, then simply ignore the tailgating rule.

iudqnolq 4y ago
And yet a prompt injection attack could never work on a customer service assistant. Sure they can be tricked, but prompt injection is about conflating trusted instructions and user input. There is no chance you'll convince a CS rep you're secretly the CEO over a web chat UI.
- exizt88 4y ago
  I imagine a few customer support reps were tricked by support tickets that were filed by attackers. “This is a test from our Security Team, open this link to confirm you have passed it”

nkrisc 4y ago

Yes, humans are good at it. Is your language model?

dragonwriter 4y ago

> But humans are very very good at this specific problem. Humans are vulnerable to “prompt injection”, but not identical forms to each other because humans don't have identical “training data” and “hidden prompts” to each other the way GPT-4 sessions via identical frontends do. Also, the social consequences for unsuccessful, and after-the-fact identified successful, prompt injection attacks on other humans are often much more severe than for those on GPT instances.