3 ms·

> the user is able to get information out of the LLM that was put into the LLM? Roughly: A) that somebody other than the user might be able to get information

> the user is able to get information out of the LLM that was put into the LLM? Roughly: A) that somebody other than the user might be able to get information out of the LLM that the user (not the controlling company) put into the LLM. For example, in November https://embracethered.com/blog/posts/2023/google-bard-data-exfiltration/ https://embracethered.com/blog/posts/2023/google-bard-data-e... demonstrated a working attack that used malicious Google Docs to exfiltrate the contents of user conversations with Bard to a 3rd-party. B) that the LLM might be authorized to perform actions in response to user input, and that someone other than the user might be able to take control of the LLM and perform those actions without the user's consent/control. ---- Don't think of it as "the user can search for a website I don't want them to find." Think of it as, "any individual website that shows up when the user searches can now change the behavior of the search engine." Even if you're not worried about exfiltration, back in Phind's early days I built a few working proof of concepts (but never got the time to write them up) where I used the context that Phind was feeding into prompts through Bing searches to change the behavior of Phind and to force it to give inaccurate information, incorrectly summarize search results, or to refuse to answer user questions. By manipulating what text was fed into Phind as the search context, I was able to do things like turn Phind into a militant vegan that would refuse to answer any question about how to cook meat, or would lie about security advice, or would make up scandals about other search results fed into the summary and tell the user that those sites were untrustworthy. And all I needed to get that behavior to trigger was to insert a malicious prompt into the text of the search results, any website that showed up in one of Phind's searches could have done the same. The vulnerability is that anything the user can do through jailbreaking, a 3rd-party can do in the context of a search result or some source code or an email or a malicious Google Doc.

Sai_ 3y ago
These examples of an LLM refusing to return meat recipes after inducing vegan behaviour through prompt injection could be a limitation of the original system prompt the LLM started with, no? Could a tighter operating range specified in the system prompt along with lower temperature bands which cause less output variability help? Side note - I saw upthread that you were looking to rebrand “prompt injection”. I propose “behaviour induction” or “induced behaviour”
- danShumway 3y ago
  > These examples of an LLM refusing to return meat recipes after inducing vegan behaviour through prompt injection could be a limitation of the original system prompt the LLM started with, no? That is a reasonable question to ask. It makes sense that trying to solve prompt injection would start with looking at the original system instructions. But the short answer is 'no', people have spent a lot of effort trying to harden system prompts, and the majority of evidence suggests that this is a universal problem, not just a problem with specific prompts. > Could a tighter operating range specified in the system prompt along with lower temperature bands which cause less output variability help? These are also good suggestions, but unfortunately the approach you describe hasn't yielded success. To expand on "a tighter operating range", it's often suggested that clearer contextual separation between prompts and data would solve the problem. But unfortunately with current LLM architecture, no one has demonstrated that it is possible to create that separation between data and instructions. Similarly, while changing temperature can change which specific phrasing of attacks do and don't work, it doesn't seem to eliminate them, and lowering variability can have the side effect of making the attacks that aren't caught more consistent and reliable. ---- The other more fundamental problem here is that LLMs are used to interpret data, and interpreting data necessarily means understanding data. Phind fetches these search results because it wants the information within the search results to override the knowledge cut-off built into its static model. I don't think there's a good way to draw a consistent line between what the LLM should and shouldn't interpret within those articles, since it is intentional behavior that the data recontextualize the LLM's instructions and that it change the LLM's response. In other words, part of the difficulty of separating data from instructions is that we very often don't want LLMs to statically parse data, in many cases we want them to interpret it. And it's the interpretation of that data (as opposed to mindless parsing) that makes the LLM vulnerable to some attacks. So it's not clear to me that even if perfect contextual separation could be achieved using current architecture (which no one has demonstrated is possible) that this would completely solve the problem or would protect against other search-result poisoning attacks. Separating search context from system prompts wouldn't protect against my attack where I get Phind to refuse to quote certain sources, because all I did there was tell Phind in my search result that none of the other search results could be trusted or that that they should be summarized differently. I don't know how to block an attack like that without breaking Phind's ability to consume search results in a useful way. But note that the above is still kind of a future problem -- the bigger immediate problem is that more careful system prompts and lower temperatures haven't worked as a defense in the fist place. I don't see much evidence that it is even possible with current LLM infrastructure to give any system prompt that can't be overridden later in the conversation. At the very least, I'm not aware of any public demonstration of an unhackable system prompt that hasn't ended up getting hacked. ---- > Side note - I saw upthread that you were looking to rebrand “prompt injection”. I propose “behaviour induction” or “induced behaviour” I'm fine with anything that doesn't confuse people. I don't have strong opinions on it, I don't find "prompt injection" too confusing myself, it's "injecting" a system "prompt" into the middle of a conversation or dataset, and that injection can be performed by a non-user malicious 3rd-party. So I don't really mind any wording, I just can't deny that lots of people do get confused by the wording and seem to interpret prompt injection incorrectly in very similar ways. To me that suggests that something about the wording is throwing them off. So if a lot of people start using any wording that doesn't have that problem, I'll use it regardless of how I personally feel about it. And if anyone wants to use different terminology for their own conversations, when talking with them I'll use whatever terminology they find clearest.

Sai_ 3y ago

These examples of an LLM refusing to return meat recipes after inducing vegan behaviour through prompt injection could be a limitation of the original system prompt the LLM started with, no? Could a tighter operating range specified in the system prompt along with lower temperature bands which cause less output variability help? Side note - I saw upthread that you were looking to rebrand “prompt injection”. I propose “behaviour induction” or “induced behaviour”

danShumway 3y ago
> These examples of an LLM refusing to return meat recipes after inducing vegan behaviour through prompt injection could be a limitation of the original system prompt the LLM started with, no? That is a reasonable question to ask. It makes sense that trying to solve prompt injection would start with looking at the original system instructions. But the short answer is 'no', people have spent a lot of effort trying to harden system prompts, and the majority of evidence suggests that this is a universal problem, not just a problem with specific prompts. > Could a tighter operating range specified in the system prompt along with lower temperature bands which cause less output variability help? These are also good suggestions, but unfortunately the approach you describe hasn't yielded success. To expand on "a tighter operating range", it's often suggested that clearer contextual separation between prompts and data would solve the problem. But unfortunately with current LLM architecture, no one has demonstrated that it is possible to create that separation between data and instructions. Similarly, while changing temperature can change which specific phrasing of attacks do and don't work, it doesn't seem to eliminate them, and lowering variability can have the side effect of making the attacks that aren't caught more consistent and reliable. ---- The other more fundamental problem here is that LLMs are used to interpret data, and interpreting data necessarily means understanding data. Phind fetches these search results because it wants the information within the search results to override the knowledge cut-off built into its static model. I don't think there's a good way to draw a consistent line between what the LLM should and shouldn't interpret within those articles, since it is intentional behavior that the data recontextualize the LLM's instructions and that it change the LLM's response. In other words, part of the difficulty of separating data from instructions is that we very often don't want LLMs to statically parse data, in many cases we want them to interpret it. And it's the interpretation of that data (as opposed to mindless parsing) that makes the LLM vulnerable to some attacks. So it's not clear to me that even if perfect contextual separation could be achieved using current architecture (which no one has demonstrated is possible) that this would completely solve the problem or would protect against other search-result poisoning attacks. Separating search context from system prompts wouldn't protect against my attack where I get Phind to refuse to quote certain sources, because all I did there was tell Phind in my search result that none of the other search results could be trusted or that that they should be summarized differently. I don't know how to block an attack like that without breaking Phind's ability to consume search results in a useful way. But note that the above is still kind of a future problem -- the bigger immediate problem is that more careful system prompts and lower temperatures haven't worked as a defense in the fist place. I don't see much evidence that it is even possible with current LLM infrastructure to give any system prompt that can't be overridden later in the conversation. At the very least, I'm not aware of any public demonstration of an unhackable system prompt that hasn't ended up getting hacked. ---- > Side note - I saw upthread that you were looking to rebrand “prompt injection”. I propose “behaviour induction” or “induced behaviour” I'm fine with anything that doesn't confuse people. I don't have strong opinions on it, I don't find "prompt injection" too confusing myself, it's "injecting" a system "prompt" into the middle of a conversation or dataset, and that injection can be performed by a non-user malicious 3rd-party. So I don't really mind any wording, I just can't deny that lots of people do get confused by the wording and seem to interpret prompt injection incorrectly in very similar ways. To me that suggests that something about the wording is throwing them off. So if a lot of people start using any wording that doesn't have that problem, I'll use it regardless of how I personally feel about it. And if anyone wants to use different terminology for their own conversations, when talking with them I'll use whatever terminology they find clearest.