4 ms·
Isn't jailbreaking a form of prompt injection, since it takes advantage of the "system" prompt being mixed together with the user prompt? I suppose there could
by ptx 3y ago
Isn't jailbreaking a form of prompt injection, since it takes advantage of the "system" prompt being mixed together with the user prompt?
I suppose there could be jailbreaks without prompt injection if the behavior is defined entirely in the fine-tuning step and there is no system prompt, but I was under the impression that ChatGPT and other services all use some kind of system prompt.
- simonw 3y agoYeah, that's part of the confusion here. Some models do indeed set some of their rules using a concatenated system prompt - but most of the "values" are baked in through instruction tuning. You can test that yourself by running local models (like Llama 2) in a context where you completely control or omit the system prompt. They will still refuse to give you bomb making recipes, or tell you how to kill Apache 2 processes (Llama 2 is notoriously sensitive in its default conditions.)
- ipython 3y agoDon't worry, we're speed running the last 50 years of computer security. What's old is now new again. Already looking at poor web application security on emerging AI/MLops tools making it rain like the 90's once again; then we have in-band signalling and lack of separation between code & data, just like back in the 70s and 80s. I totally get your frustration, it's because you've seen the pattern before. Enjoy the ride as we all rediscover these fundamental truths we learned decades ago!
- mindcrime 3y agoSay hello to BlueBoxGPT and the new era of "llm phreaking"!