4 ms·

As long as your evil prompt is not permanently changing the LLM, this is harmless. If you want to know how to do <bad thing>, the information is out there. You

As long as your evil prompt is not permanently changing the LLM, this is harmless. If you want to know how to do <bad thing>, the information is out there. You can trick an LLM into giving it to you, so what? One commenter says it could be harmful when LLMs are used for something important, like medical diagnosis. However, I don't see a healthcare practitioner using evil suffixes. And if they do, that's on them, just another form of malpractice. People need to understand that LLMs are just fancy statistical tables, generating random stuff from their training data. All the angst about generating undesirable random stuff is just silly...

BasedAnon 3y ago
It means that any LLM that is public facing can be used for arbitrary needs, regardless what it was initially prompted with. Picture for example, someone writing an API to jailbreak a support system so they can use it as a free translator with no usage limits.
yeck 3y ago
If you spend timing considering how to use LLMs as part of another product or system you realize quickly that there are a lot of interesting things that you just can safely do because you can't control the LLMs inputs or outputs. I have personally examined use cases for products that would be used by children and I cannot use LLMs as it currently stands because I have no way to ensure the content generated is age appropriate.
- bick_nyers 3y ago
  Couldn't you check the response before you send it to the user? I'm sure there's tons of keywords you could check. You could also run the response itself through an LLM to characterize it.
  - jstarfish 3y ago
    ChatGPT currently does this with a post-render Javascript call. It'll blank out your input and redact whatever it rendered.
danShumway 3y ago
These attacks target jailbreaking because that's the easiest thing to test, but when you're reading about them, you really need to mentally substitute "jailbreak" with "indirect prompt injection." https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ For the most part you're right; there are reasons why you'd want to minimize jailbreaks but it's not the end of the world if you can get a model to swear. But if you can't keep a model from swearing, then you also can't keep it from phishing the user based on instructions in a PDF it reads. It's reasonable that researchers target jailbreaking because it's kind of the low-hanging fruit of LLM defense, it's likely easier to defend against a jailbreak than against general malicious input. And like I mentioned above, it's much easier to test as well. But it's unfortunate that a consequence of focusing on jailbreaking is that it creates the perception that jailbreaks are the only thing that's being tested.

BasedAnon 3y ago

It means that any LLM that is public facing can be used for arbitrary needs, regardless what it was initially prompted with. Picture for example, someone writing an API to jailbreak a support system so they can use it as a free translator with no usage limits.

yeck 3y ago

If you spend timing considering how to use LLMs as part of another product or system you realize quickly that there are a lot of interesting things that you just can safely do because you can't control the LLMs inputs or outputs. I have personally examined use cases for products that would be used by children and I cannot use LLMs as it currently stands because I have no way to ensure the content generated is age appropriate.

bick_nyers 3y ago
Couldn't you check the response before you send it to the user? I'm sure there's tons of keywords you could check. You could also run the response itself through an LLM to characterize it.
- jstarfish 3y ago
  ChatGPT currently does this with a post-render Javascript call. It'll blank out your input and redact whatever it rendered.

danShumway 3y ago

These attacks target jailbreaking because that's the easiest thing to test, but when you're reading about them, you really need to mentally substitute "jailbreak" with "indirect prompt injection." https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ For the most part you're right; there are reasons why you'd want to minimize jailbreaks but it's not the end of the world if you can get a model to swear. But if you can't keep a model from swearing, then you also can't keep it from phishing the user based on instructions in a PDF it reads. It's reasonable that researchers target jailbreaking because it's kind of the low-hanging fruit of LLM defense, it's likely easier to defend against a jailbreak than against general malicious input. And like I mentioned above, it's much easier to test as well. But it's unfortunate that a consequence of focusing on jailbreaking is that it creates the perception that jailbreaks are the only thing that's being tested.