3 ms·

I was going to assert that the 'system role' provided by the API should prevent this problem if used properly. But then I stumbled this recent information, whi

I was going to assert that the 'system role' provided by the API should prevent this problem if used properly. But then I stumbled this recent information, which seems to say that the 'system role' is not quite behaving as intended or as you might expect from reading the docs. https://community.openai.com/t/the-system-role-how-it-influences-the-chat-behavior/87353 https://community.openai.com/t/the-system-role-how-it-influe...

simonw 3y ago
I wrote a bit about the system prompt here - it's better in GPT-4 but can still be defeated by a devious enough attack prompt: https://simonwillison.net/2023/Apr/14/worst-that-can-happen/#gpt4 https://simonwillison.net/2023/Apr/14/worst-that-can-happen/...
- squeaky-clean 3y ago
  Have you managed to break recent versions of Bing Chat? I've tried some of the notable injection prompts and my own variants. Sometimes I can get it to begin with the jailbreak, but after 10-20 words it will erase the previous text and say something along the lines of "I cannot talk about that"
  - codeflo 3y ago
    My reading of these events is that the jailbreak is successful, but some other, slower system detects it after the fact and deletes the message. Given the delay, I've seen speculation that there's a manual step involved.
  - simonw 3y ago
    Jailbreaking and prompt injection are related but they aren't quite the same problem. That thing where Bing redacts text after the point isn't going to protect your application against problems like those I describe in https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ https://simonwillison.net/2023/Apr/14/worst-that-can-happen/
    - squeaky-clean 3y ago
      Great point
cmelbye 3y ago
They didn’t promise that “system” fixes the problem. It is a prerequisite to a possible solution for the problem. Now that the model knows which messages are “system” and “user”, it can be trained to treat them differently.

simonw 3y ago

I wrote a bit about the system prompt here - it's better in GPT-4 but can still be defeated by a devious enough attack prompt: https://simonwillison.net/2023/Apr/14/worst-that-can-happen/#gpt4 https://simonwillison.net/2023/Apr/14/worst-that-can-happen/...

squeaky-clean 3y ago
Have you managed to break recent versions of Bing Chat? I've tried some of the notable injection prompts and my own variants. Sometimes I can get it to begin with the jailbreak, but after 10-20 words it will erase the previous text and say something along the lines of "I cannot talk about that"
- codeflo 3y ago
  My reading of these events is that the jailbreak is successful, but some other, slower system detects it after the fact and deletes the message. Given the delay, I've seen speculation that there's a manual step involved.
- simonw 3y ago
  Jailbreaking and prompt injection are related but they aren't quite the same problem. That thing where Bing redacts text after the point isn't going to protect your application against problems like those I describe in https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ https://simonwillison.net/2023/Apr/14/worst-that-can-happen/
  - squeaky-clean 3y ago
    Great point

cmelbye 3y ago

They didn’t promise that “system” fixes the problem. It is a prerequisite to a possible solution for the problem. Now that the model knows which messages are “system” and “user”, it can be trained to treat them differently.