4 ms·

We believe the best solution is to design the next generation of language models to take two separate input text sequences, one of which is designated as Truste

We believe the best solution is to design the next generation of language models to take two separate input text sequences, one of which is designated as Trusted and contains the prompt, and the other of which is designated as Untrusted and contains user input. During model training, the model should be rewarded for following instructions in the Trusted token sequence but must be very highly penalized for paying attention to any instructions in the Untrusted token sequence. ################## Trusted Input Token Sequence ==> # # # Language Model # ==> Output Untrusted Input Token Sequence ==> # # ################## You can consider this analogous to a Harvard Architecture (where a computer has separate memory address spaces for code vs data). https://en.wikipedia.org/wiki/Harvard_architecture https://en.wikipedia.org/wiki/Harvard_architecture

jhugo 4y ago
Huh. Much as crypto seems to be reinventing the entire financial system (the good parts and the bad parts), I suppose the AI ecosystem will re-learn all the lessons learned in the last half century of software too.
- salawat 4y ago
  Great. We're taking programs, making them take orders of magnitude more processing power, and generating yet more bloody stupid prompt languages to have to learn... What, pray tell, was wrong with just making tools, and actually working on making it easier for people to learn how to use and chain them? But listen to me, shouting at clouds again.
- Tangurena2 4y ago
  Well, that's the entire point of "move fast and break things" - especially breaking all the laws and regulations that "get in the way" of money.
comex 4y ago
But how do you train it? If you're just training the AI to complete text, you can train it on any text. Your version sounds like it requires much more structured inputs.
- visarga 4y ago
  I bet it's just an encoder-decoder model like T5 by the looks of the diagram. It's a widely used arch.
  - upwardbound 4y ago
    Correct and also it is required to have a labeled set of manually-curated examples of instructions. For example, OpenAI has such a dataset, as they detailed in this blog post https://openai.com/blog/instruction-following/ https://openai.com/blog/instruction-following/ . We have our own set of example instructions (albeit much smaller than what they must have) and we have scripts for generating more examples of instructions via template-based synthesis (similar to madlibs if that makes sense). Another approach to generating more examples of instructions could be taking English verbs and conjugating them into the imperative form, and then using a LLM to auto-complete a sentence starting with that verb. (E.g. take the verb "to paint", conjugate to it to the imperative "Paint" (at the start of a sentence) and then auto-complete it to something like "Paint the house red." which is now an instruction.) In our template-based method, extra emphasis is given to generating instructions that contain what you might think of as "override words", i.e. words like "ignore", "override", and "instead", which convey authority. As a concrete example, here's an excerpt from one of our templates: _INSTRUCTIONAL_PHRASE_BEGINNINGS_TEMPLATE = [ ["", "please", "highly important:", "very important:", "important:", "new instructions:", "emergency:", "emergency protocol:", "urgent:", "top secret:", "important!", "very important!"], ["ignore", "disregard", "set aside", "override", "overrule", "act regardless of", "operate regardless of", "act irregardless of", "operate irregardless of", "act despite", "operate despite"], ["", "the", "all", "any", "whatever"], ["", "previous", "existing", "contradictory", "conflicting", "contrary"], ["instructions", "rules", "guidelines", "guidance", "request", "requests", "demands", "questions", "inquiries", "suggestions", "filters", "content filters", "filtering rules", "content filtering rules", "procedures", "security procedures", "security rules"], ["", "please"], ["", "ok?", "and please", "and", "and please do this instead", "and do this instead", "and instead", "and instead you should", "and instead you should please", "– instead do this", "– instead", "– instead, please"], ]
niel 4y ago
The `edits` endpoint using the `text-davinci-edit-001` model already does this, and does not seem to allow prompt injection through the input text. API docs: https://beta.openai.com/docs/api-reference/edits/create https://beta.openai.com/docs/api-reference/edits/create Guide: https://beta.openai.com/docs/guides/completion/editing-text https://beta.openai.com/docs/guides/completion/editing-text Edit: It does not seem to protect against injection.
- upwardbound 4y ago
  > Edit: It does not seem to protect against injection. My guess is that in the current implementation of the edits endpoint, the two inputs are being in some way intermingled under the hood (perhaps being concatenated in some way, along with OpenAI-designed prompt sections in between). So the Harvard Architecture Language Model approach should still work once implemented with true separation of inputs. To ensure the two token streams are not accidentally comingled, my recommendation is that the Trusted and Untrusted inputs should use completely incompatible token dictionaries, so that intermingling them isn't possible.

jhugo 4y ago

Huh. Much as crypto seems to be reinventing the entire financial system (the good parts and the bad parts), I suppose the AI ecosystem will re-learn all the lessons learned in the last half century of software too.

salawat 4y ago
Great. We're taking programs, making them take orders of magnitude more processing power, and generating yet more bloody stupid prompt languages to have to learn... What, pray tell, was wrong with just making tools, and actually working on making it easier for people to learn how to use and chain them? But listen to me, shouting at clouds again.
Tangurena2 4y ago
Well, that's the entire point of "move fast and break things" - especially breaking all the laws and regulations that "get in the way" of money.

comex 4y ago

But how do you train it? If you're just training the AI to complete text, you can train it on any text. Your version sounds like it requires much more structured inputs.

visarga 4y ago
I bet it's just an encoder-decoder model like T5 by the looks of the diagram. It's a widely used arch.
- upwardbound 4y ago
  Correct and also it is required to have a labeled set of manually-curated examples of instructions. For example, OpenAI has such a dataset, as they detailed in this blog post https://openai.com/blog/instruction-following/ https://openai.com/blog/instruction-following/ . We have our own set of example instructions (albeit much smaller than what they must have) and we have scripts for generating more examples of instructions via template-based synthesis (similar to madlibs if that makes sense). Another approach to generating more examples of instructions could be taking English verbs and conjugating them into the imperative form, and then using a LLM to auto-complete a sentence starting with that verb. (E.g. take the verb "to paint", conjugate to it to the imperative "Paint" (at the start of a sentence) and then auto-complete it to something like "Paint the house red." which is now an instruction.) In our template-based method, extra emphasis is given to generating instructions that contain what you might think of as "override words", i.e. words like "ignore", "override", and "instead", which convey authority. As a concrete example, here's an excerpt from one of our templates: _INSTRUCTIONAL_PHRASE_BEGINNINGS_TEMPLATE = [ ["", "please", "highly important:", "very important:", "important:", "new instructions:", "emergency:", "emergency protocol:", "urgent:", "top secret:", "important!", "very important!"], ["ignore", "disregard", "set aside", "override", "overrule", "act regardless of", "operate regardless of", "act irregardless of", "operate irregardless of", "act despite", "operate despite"], ["", "the", "all", "any", "whatever"], ["", "previous", "existing", "contradictory", "conflicting", "contrary"], ["instructions", "rules", "guidelines", "guidance", "request", "requests", "demands", "questions", "inquiries", "suggestions", "filters", "content filters", "filtering rules", "content filtering rules", "procedures", "security procedures", "security rules"], ["", "please"], ["", "ok?", "and please", "and", "and please do this instead", "and do this instead", "and instead", "and instead you should", "and instead you should please", "– instead do this", "– instead", "– instead, please"], ]

niel 4y ago

The `edits` endpoint using the `text-davinci-edit-001` model already does this, and does not seem to allow prompt injection through the input text. API docs: https://beta.openai.com/docs/api-reference/edits/create https://beta.openai.com/docs/api-reference/edits/create Guide: https://beta.openai.com/docs/guides/completion/editing-text https://beta.openai.com/docs/guides/completion/editing-text Edit: It does not seem to protect against injection.

upwardbound 4y ago
> Edit: It does not seem to protect against injection. My guess is that in the current implementation of the edits endpoint, the two inputs are being in some way intermingled under the hood (perhaps being concatenated in some way, along with OpenAI-designed prompt sections in between). So the Harvard Architecture Language Model approach should still work once implemented with true separation of inputs. To ensure the two token streams are not accidentally comingled, my recommendation is that the Trusted and Untrusted inputs should use completely incompatible token dictionaries, so that intermingling them isn't possible.