3 ms·

What are ANY consequential impacts of prompt injection other that the user is able to get information out of the LLM that was put into the LLM? I can not under

What are ANY consequential impacts of prompt injection other that the user is able to get information out of the LLM that was put into the LLM? I can not understand what the concern is. Like if something is indexed by Google, that means it might be available to find through a search, same with an LLM.

dragonwriter 3y ago
> What are ANY consequential impacts of prompt injection other that the user is able to get information out of the LLM that was put into the LLM? The impact of prompt injection is provoking arbitrary, unintended behavior from the LLM. If the LLM is a simple chatbot with no tool use beyond retrieving data, that just means “retrieving data different than the LLM operator would have anticipated” (and possibly the user—prompt injection can be done by data retrieved that the user doesn't control, not just the user themselves, because all data processed by the LLM passes through as part of a prompt). But if the LLM is tied into a framework where it serves as an agent with active tool use, then the blast radius of prompt injection is much bigger. A lot of the concern about prompt injection isn't about currently popular applications of LLMs, but the applications that have been set out as near term possibilities that are much more powerful.
- simonw 3y ago
  Exactly this. Prompt injection severity varies depending on the application. The biggest risk come from applications that have tool access, but applications that can access private data have a risk too thanks to various data exfiltration tricks.
simonw 3y ago
I've written a bunch about this: - Prompt injection: What’s the worst that can happen? https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ - The Dual LLM pattern for building AI assistants that can resist prompt injection https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ - Prompt injection explained, November 2023 edition https://simonwillison.net/2023/Nov/27/prompt-injection-explained/ https://simonwillison.net/2023/Nov/27/prompt-injection-expla... More here: https://simonwillison.net/series/prompt-injection/ https://simonwillison.net/series/prompt-injection/
danShumway 3y ago
> the user is able to get information out of the LLM that was put into the LLM? Roughly: A) that somebody other than the user might be able to get information out of the LLM that the user (not the controlling company) put into the LLM. For example, in November https://embracethered.com/blog/posts/2023/google-bard-data-exfiltration/ https://embracethered.com/blog/posts/2023/google-bard-data-e... demonstrated a working attack that used malicious Google Docs to exfiltrate the contents of user conversations with Bard to a 3rd-party. B) that the LLM might be authorized to perform actions in response to user input, and that someone other than the user might be able to take control of the LLM and perform those actions without the user's consent/control. ---- Don't think of it as "the user can search for a website I don't want them to find." Think of it as, "any individual website that shows up when the user searches can now change the behavior of the search engine." Even if you're not worried about exfiltration, back in Phind's early days I built a few working proof of concepts (but never got the time to write them up) where I used the context that Phind was feeding into prompts through Bing searches to change the behavior of Phind and to force it to give inaccurate information, incorrectly summarize search results, or to refuse to answer user questions. By manipulating what text was fed into Phind as the search context, I was able to do things like turn Phind into a militant vegan that would refuse to answer any question about how to cook meat, or would lie about security advice, or would make up scandals about other search results fed into the summary and tell the user that those sites were untrustworthy. And all I needed to get that behavior to trigger was to insert a malicious prompt into the text of the search results, any website that showed up in one of Phind's searches could have done the same. The vulnerability is that anything the user can do through jailbreaking, a 3rd-party can do in the context of a search result or some source code or an email or a malicious Google Doc.
- Sai_ 3y ago
  These examples of an LLM refusing to return meat recipes after inducing vegan behaviour through prompt injection could be a limitation of the original system prompt the LLM started with, no? Could a tighter operating range specified in the system prompt along with lower temperature bands which cause less output variability help? Side note - I saw upthread that you were looking to rebrand “prompt injection”. I propose “behaviour induction” or “induced behaviour”

dragonwriter 3y ago

> What are ANY consequential impacts of prompt injection other that the user is able to get information out of the LLM that was put into the LLM? The impact of prompt injection is provoking arbitrary, unintended behavior from the LLM. If the LLM is a simple chatbot with no tool use beyond retrieving data, that just means “retrieving data different than the LLM operator would have anticipated” (and possibly the user—prompt injection can be done by data retrieved that the user doesn't control, not just the user themselves, because all data processed by the LLM passes through as part of a prompt). But if the LLM is tied into a framework where it serves as an agent with active tool use, then the blast radius of prompt injection is much bigger. A lot of the concern about prompt injection isn't about currently popular applications of LLMs, but the applications that have been set out as near term possibilities that are much more powerful.

simonw 3y ago
Exactly this. Prompt injection severity varies depending on the application. The biggest risk come from applications that have tool access, but applications that can access private data have a risk too thanks to various data exfiltration tricks.

simonw 3y ago

I've written a bunch about this: - Prompt injection: What’s the worst that can happen? https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ - The Dual LLM pattern for building AI assistants that can resist prompt injection https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ - Prompt injection explained, November 2023 edition https://simonwillison.net/2023/Nov/27/prompt-injection-explained/ https://simonwillison.net/2023/Nov/27/prompt-injection-expla... More here: https://simonwillison.net/series/prompt-injection/ https://simonwillison.net/series/prompt-injection/

danShumway 3y ago

> the user is able to get information out of the LLM that was put into the LLM? Roughly: A) that somebody other than the user might be able to get information out of the LLM that the user (not the controlling company) put into the LLM. For example, in November https://embracethered.com/blog/posts/2023/google-bard-data-exfiltration/ https://embracethered.com/blog/posts/2023/google-bard-data-e... demonstrated a working attack that used malicious Google Docs to exfiltrate the contents of user conversations with Bard to a 3rd-party. B) that the LLM might be authorized to perform actions in response to user input, and that someone other than the user might be able to take control of the LLM and perform those actions without the user's consent/control. ---- Don't think of it as "the user can search for a website I don't want them to find." Think of it as, "any individual website that shows up when the user searches can now change the behavior of the search engine." Even if you're not worried about exfiltration, back in Phind's early days I built a few working proof of concepts (but never got the time to write them up) where I used the context that Phind was feeding into prompts through Bing searches to change the behavior of Phind and to force it to give inaccurate information, incorrectly summarize search results, or to refuse to answer user questions. By manipulating what text was fed into Phind as the search context, I was able to do things like turn Phind into a militant vegan that would refuse to answer any question about how to cook meat, or would lie about security advice, or would make up scandals about other search results fed into the summary and tell the user that those sites were untrustworthy. And all I needed to get that behavior to trigger was to insert a malicious prompt into the text of the search results, any website that showed up in one of Phind's searches could have done the same. The vulnerability is that anything the user can do through jailbreaking, a 3rd-party can do in the context of a search result or some source code or an email or a malicious Google Doc.

Sai_ 3y ago
These examples of an LLM refusing to return meat recipes after inducing vegan behaviour through prompt injection could be a limitation of the original system prompt the LLM started with, no? Could a tighter operating range specified in the system prompt along with lower temperature bands which cause less output variability help? Side note - I saw upthread that you were looking to rebrand “prompt injection”. I propose “behaviour induction” or “induced behaviour”