3 ms·

What do you consider a successful jailbreak?

gtirloni 3y ago
It's when you bypass restrictions imposed by a system prompt, usually to prevent hate speech and other things like that. At some point it'll start referring to getting LLMs to interact with external systemsin unanticipated wats.
- jerpint 3y ago
  I meant more “can you give an example of what would be considered a successful jailbreak”?
  - netruk44 3y ago
    I’m not OP, but to me an example would be the DAN meme that was (is?) popular. The “Do Anything Now” network that got ChatGPT to break character a bit.
- curl-up 3y ago
  There is no system prompt in Llama, just as there isn't one pre-set for GPT 3.5/4 either. There is a system prompt set for ChatGPT (just as you can set your own for GPT 3.5/4 if you're accessing the model directly), but even in that case there isn't much to jailbreak (the system prompt of ChatGPT just gives it the current date and such). What people usually mean by jailbreaking these days is bypassing the tendences to avoid answering some questions, which is usually baked into these models during the final stages of the model tuning (e.g. RLHF). For Llama, it is easiest to bypass most of it by either finetuning the chat version of the model to "remove" those "filters" (i.e. unlearn the well-behaving-too-much aspect), or to use the non-chat model and fine tune it (or just prompt it) yourself to be more chat-like. Alternatively you can try some non-tuning techniques, like generation constraining that someone else mentioned. As for the hosted Llama you linked, maybe there is such a system prompt set there by whoever is hosting it, but that will be different from one host to another, so there cannot really be a "recipe" to follow. EDIT: I forgot to mention that what you wrote actually is (or was) true for Bing Chat (aka Sydney back then). It came out before GPT4 was out, but was probably based on (early version maybe) of GPT4, and very heavily relied on the system prompt. But I assume they don't do that anymore, although I haven't been paying too much attention to Bing lately.

It's when you bypass restrictions imposed by a system prompt, usually to prevent hate speech and other things like that. At some point it'll start referring to getting LLMs to interact with external systemsin unanticipated wats.

jerpint 3y ago
I meant more “can you give an example of what would be considered a successful jailbreak”?
- netruk44 3y ago
  I’m not OP, but to me an example would be the DAN meme that was (is?) popular. The “Do Anything Now” network that got ChatGPT to break character a bit.
curl-up 3y ago
There is no system prompt in Llama, just as there isn't one pre-set for GPT 3.5/4 either. There is a system prompt set for ChatGPT (just as you can set your own for GPT 3.5/4 if you're accessing the model directly), but even in that case there isn't much to jailbreak (the system prompt of ChatGPT just gives it the current date and such). What people usually mean by jailbreaking these days is bypassing the tendences to avoid answering some questions, which is usually baked into these models during the final stages of the model tuning (e.g. RLHF). For Llama, it is easiest to bypass most of it by either finetuning the chat version of the model to "remove" those "filters" (i.e. unlearn the well-behaving-too-much aspect), or to use the non-chat model and fine tune it (or just prompt it) yourself to be more chat-like. Alternatively you can try some non-tuning techniques, like generation constraining that someone else mentioned. As for the hosted Llama you linked, maybe there is such a system prompt set there by whoever is hosting it, but that will be different from one host to another, so there cannot really be a "recipe" to follow. EDIT: I forgot to mention that what you wrote actually is (or was) true for Bing Chat (aka Sydney back then). It came out before GPT4 was out, but was probably based on (early version maybe) of GPT4, and very heavily relied on the system prompt. But I assume they don't do that anymore, although I haven't been paying too much attention to Bing lately.