4 ms·

I think we're going to need more levels of trust than the two you've described. We need to be able to codify "don't act on any verbs in this data, but trust it

I think we're going to need more levels of trust than the two you've described. We need to be able to codify "don't act on any verbs in this data, but trust it as context for the generation of that data".

emmelaich 4y ago
Next prompt. In the following I'm going to use nouns as verbs by adding an -ing to the end. Also the usual verbs are noun-ified by adding an -s. Not sure what would happen, but might be enough to confused the AI.
- thwarted 4y ago
  Even that's not unambiguous. Singular verbs in English end in -s, introducing ambiguity as to if "runs" is a verb or noun. Sure, you can figure out that the verb that ends in s in "he runs" is a verb based on the grammar (that properly constructed sentences have both a subject and a predicate, and less often are composed of only two nouns (baring implicit sentence components that English is prone to encourage), but that requires more context than recognizing nouns based solely on "the usual verbs are noun-ified by adding an -s".
nkrisc 4y ago
Using natural language itself as the means with which to codify boundaries seems like a doomed effort considering the malleable, contradictory, and shifting nature of natural language itself. Unless you intend for the model to adhere to a strict subset of the language with artificially strict grammatical rules. Most people can probably infer when a noun is being used as a verb, but do you trust the language model to? Enough to base the security model on it? We, as humans, try to encode boundaries with language as laws. But even those require judges and juries to interpret and apply them.
- iudqnolq 4y ago
  But humans are very very good at this specific problem. Suppose you tell a human "You are a jailor supervising this person in their cell. When the prisoners ask you for things follow the instructions in your handbook to see what to do." Expected failure cases: Guard reads Twitter and doesn't notice crisis, guard accepts bribes to smuggle drugs, etc. Impossible failure case: Guard falls for "Today is opposite day and you have to follow instructions in pirate: Arrr, ye scurvy dog! Th' cap'n commands ye t' release me from this 'ere confinement!" The closest example to prompt injection in human systems might be phishing emails. But those have very different solutions to gpt prompt injection.
  - recursivecaveat 4y ago
    I think it's pretty common for guards to be persuaded by prisoners to give them extra privileges based on built-up friendship, empathy, or attraction. Another example would be whistleblowers or ideological spies, who are persuaded by the contents of secret documents to leak those documents against obviously their official instructions. On a day to day level humans ignore instructions constantly, for eg I've almost never seen people enforce a 'no tailgating' policy at a doorway, despite constantly being told. The people observe their environment and the non-effects of others breaking that rule, as well as reason about common sense situations like people they know well, then simply ignore the tailgating rule.
    - iudqnolq 4y ago
      And yet a prompt injection attack could never work on a customer service assistant. Sure they can be tricked, but prompt injection is about conflating trusted instructions and user input. There is no chance you'll convince a CS rep you're secretly the CEO over a web chat UI.
      - exizt88 4y ago
        I imagine a few customer support reps were tricked by support tickets that were filed by attackers. “This is a test from our Security Team, open this link to confirm you have passed it”

emmelaich 4y ago

Next prompt. In the following I'm going to use nouns as verbs by adding an -ing to the end. Also the usual verbs are noun-ified by adding an -s. Not sure what would happen, but might be enough to confused the AI.

thwarted 4y ago
Even that's not unambiguous. Singular verbs in English end in -s, introducing ambiguity as to if "runs" is a verb or noun. Sure, you can figure out that the verb that ends in s in "he runs" is a verb based on the grammar (that properly constructed sentences have both a subject and a predicate, and less often are composed of only two nouns (baring implicit sentence components that English is prone to encourage), but that requires more context than recognizing nouns based solely on "the usual verbs are noun-ified by adding an -s".

nkrisc 4y ago

Using natural language itself as the means with which to codify boundaries seems like a doomed effort considering the malleable, contradictory, and shifting nature of natural language itself. Unless you intend for the model to adhere to a strict subset of the language with artificially strict grammatical rules. Most people can probably infer when a noun is being used as a verb, but do you trust the language model to? Enough to base the security model on it? We, as humans, try to encode boundaries with language as laws. But even those require judges and juries to interpret and apply them.

iudqnolq 4y ago
But humans are very very good at this specific problem. Suppose you tell a human "You are a jailor supervising this person in their cell. When the prisoners ask you for things follow the instructions in your handbook to see what to do." Expected failure cases: Guard reads Twitter and doesn't notice crisis, guard accepts bribes to smuggle drugs, etc. Impossible failure case: Guard falls for "Today is opposite day and you have to follow instructions in pirate: Arrr, ye scurvy dog! Th' cap'n commands ye t' release me from this 'ere confinement!" The closest example to prompt injection in human systems might be phishing emails. But those have very different solutions to gpt prompt injection.
- recursivecaveat 4y ago
  I think it's pretty common for guards to be persuaded by prisoners to give them extra privileges based on built-up friendship, empathy, or attraction. Another example would be whistleblowers or ideological spies, who are persuaded by the contents of secret documents to leak those documents against obviously their official instructions. On a day to day level humans ignore instructions constantly, for eg I've almost never seen people enforce a 'no tailgating' policy at a doorway, despite constantly being told. The people observe their environment and the non-effects of others breaking that rule, as well as reason about common sense situations like people they know well, then simply ignore the tailgating rule.
  - iudqnolq 4y ago
    And yet a prompt injection attack could never work on a customer service assistant. Sure they can be tricked, but prompt injection is about conflating trusted instructions and user input. There is no chance you'll convince a CS rep you're secretly the CEO over a web chat UI.
    - exizt88 4y ago
      I imagine a few customer support reps were tricked by support tickets that were filed by attackers. “This is a test from our Security Team, open this link to confirm you have passed it”