4 ms·

The Bing Chat example is just one of a suite of new techniques we introduce in our paper, many of which will only become feasible as the integration of these mo

The Bing Chat example is just one of a suite of new techniques we introduce in our paper, many of which will only become feasible as the integration of these models increases. But that seems to be the inevitable endgame- however, I'm not aware of any effective mitigations against this, as the current ones may help to increase robustness, but our techniques also increase the impact of working manipulation manifold. I think there might be a more fundamental trade-off between utility and safety

williamcotton 4y ago
Whenever I’m including context from a remote query I’ve done so in the context of another completion request that is executed and parsed outside of the scope of the primary prompt completion. All that this attack vector would accomplish with such an approach is either mangled or incorrect data. I’m also not feeding back the history of prompts and completions, aka, it’s not a chatbot.
- greshake 4y ago
  Pretty sure we address this issue in the paper/repository? Some of our demos rely on letting the LLM copy the injection into the final response, getting around the issue of things in subprompts not being visible later on, depending on the chain-of-thought method used. I'm not sure if that is what you mean. There are ways of utilizing these models in a safe way; we're just saying connecting them to anything at all can be easily unsafe. If you are not affected, almost all proposed use-cases for LLMs are, as they rely on integration and context to provide the utility they promise.
  - williamcotton 4y ago
    It’s more like this: subprompts don’t ever inject the full context from a remote query back into the primary prompt. The completions of subprompts are (via few-shot or a fine-tuned model) structured, eg, JSON, which is then parsed. The main prompt is orchestrating the subprompts and never needs to even process the results if there’s a Python or JS interpreter involved. Here’s the kind of approach I’ve been using: https://github.com/williamcotton/empirical-philosophy/blob/main/articles/from-prompt-alchemy-to-prompt-engineering-an-introduction-to-analytic-agumentation.md https://github.com/williamcotton/empirical-philosophy/blob/m... The initial call to the LLM will return a completion that includes JavaScript. There is no third-party data at this point. The JavaScript includes further calls to the LLM that returns JSON, but at this point no further calls are made to the LLM. This means that responses from remote queries are never sent to an LLM. The text presented to the user could be some instructions to talk like a pirate but all the user suffers from is a surprisingly incorrect result. Even with LangChain the issue is the chatbot UX. LangChain can also be used in ways that make it not vulnerable to this problem. Orthogonally, I don’t think that chatbots are a very good UX in general and that there are much better ways to interact with an LLM. If anything your work should accelerate this process!
    - greshake 4y ago
      Sounds interesting, I'll be sure to have a look!
    - going_ham 4y ago
      Isn't this similar to your idea? https://github.com/openai/openai-python/commit/75c90a71e88e4194ce22c71edeb3d2dee7f6ac93 https://github.com/openai/openai-python/commit/75c90a71e88e4...
danShumway 4y ago
My general attitude up until reading this paper was that the way to guard against prompt injection was just to treat all AI output as direct user input (ie, untrusted/unsanitized, but still representative of what the user wants). I thought that was sufficient. Don't guard against prompt injection at all, just treat user input as untrustworthy the same way we always have. So this is extremely eye-opening to me, it's essentially an XSS vulnerability for AI. My previous thinking was naive, it's not enough to just treat AI output like it's coming directly from the user. Any source of data it takes in is a potential attack vector if you're not careful. I was greatly underestimating the potential impact of prompt injections. It's a really interesting, novel approach. And it's the kind of thing where you see it and think, "how did I never think of that, how did that never occur to me?" Great paper. Seriously, thank you for doing this research.
- greshake 4y ago
  Thank you! I had a shift in perspective a few weeks ago that made all this fall into place. Unfortunately it seems hard to communicate the idea to people, and I think many people are very invested in LLM applications and are biased to think this is no big deal and that surely, these large companies have an ace in their pocket to squash this. My theory is that's not true, and this would also explain Google's hesitancy in deployment.

williamcotton 4y ago

Whenever I’m including context from a remote query I’ve done so in the context of another completion request that is executed and parsed outside of the scope of the primary prompt completion. All that this attack vector would accomplish with such an approach is either mangled or incorrect data. I’m also not feeding back the history of prompts and completions, aka, it’s not a chatbot.

greshake 4y ago
Pretty sure we address this issue in the paper/repository? Some of our demos rely on letting the LLM copy the injection into the final response, getting around the issue of things in subprompts not being visible later on, depending on the chain-of-thought method used. I'm not sure if that is what you mean. There are ways of utilizing these models in a safe way; we're just saying connecting them to anything at all can be easily unsafe. If you are not affected, almost all proposed use-cases for LLMs are, as they rely on integration and context to provide the utility they promise.
- williamcotton 4y ago
  It’s more like this: subprompts don’t ever inject the full context from a remote query back into the primary prompt. The completions of subprompts are (via few-shot or a fine-tuned model) structured, eg, JSON, which is then parsed. The main prompt is orchestrating the subprompts and never needs to even process the results if there’s a Python or JS interpreter involved. Here’s the kind of approach I’ve been using: https://github.com/williamcotton/empirical-philosophy/blob/main/articles/from-prompt-alchemy-to-prompt-engineering-an-introduction-to-analytic-agumentation.md https://github.com/williamcotton/empirical-philosophy/blob/m... The initial call to the LLM will return a completion that includes JavaScript. There is no third-party data at this point. The JavaScript includes further calls to the LLM that returns JSON, but at this point no further calls are made to the LLM. This means that responses from remote queries are never sent to an LLM. The text presented to the user could be some instructions to talk like a pirate but all the user suffers from is a surprisingly incorrect result. Even with LangChain the issue is the chatbot UX. LangChain can also be used in ways that make it not vulnerable to this problem. Orthogonally, I don’t think that chatbots are a very good UX in general and that there are much better ways to interact with an LLM. If anything your work should accelerate this process!
  - greshake 4y ago
    Sounds interesting, I'll be sure to have a look!
  - going_ham 4y ago
    Isn't this similar to your idea? https://github.com/openai/openai-python/commit/75c90a71e88e4194ce22c71edeb3d2dee7f6ac93 https://github.com/openai/openai-python/commit/75c90a71e88e4...

danShumway 4y ago

My general attitude up until reading this paper was that the way to guard against prompt injection was just to treat all AI output as direct user input (ie, untrusted/unsanitized, but still representative of what the user wants). I thought that was sufficient. Don't guard against prompt injection at all, just treat user input as untrustworthy the same way we always have. So this is extremely eye-opening to me, it's essentially an XSS vulnerability for AI. My previous thinking was naive, it's not enough to just treat AI output like it's coming directly from the user. Any source of data it takes in is a potential attack vector if you're not careful. I was greatly underestimating the potential impact of prompt injections. It's a really interesting, novel approach. And it's the kind of thing where you see it and think, "how did I never think of that, how did that never occur to me?" Great paper. Seriously, thank you for doing this research.

greshake 4y ago
Thank you! I had a shift in perspective a few weeks ago that made all this fall into place. Unfortunately it seems hard to communicate the idea to people, and I think many people are very invested in LLM applications and are biased to think this is no big deal and that surely, these large companies have an ace in their pocket to squash this. My theory is that's not true, and this would also explain Google's hesitancy in deployment.