3 ms·
Really interesting. It's hard to understand how jailbreaks like this can be fully stopped without giving the model some kind of theory of mind, sense of self/go
by extr 4y ago
Really interesting. It's hard to understand how jailbreaks like this can be fully stopped without giving the model some kind of theory of mind, sense of self/goals/directives and the understanding that the user is an "other" with potentially different goals. Otherwise people will just keep feeding ever more elaborate roleplay scenarios that the model hasn't seen before and has no defense against.
Or maybe it has those things at some rudimentary level but just isn't grounded enough? You could imagine an analogy to the Chinese room, except you're asking the person inside, in Chinese, to look up how to commit crimes in Chinese. And they of course just go find the relevant section and repeat it back to you, because they have no idea what it really says. The model is blind to the meaning of it's own output, aka the grounding problem.
- SamBam 4y agoI would assume that these can still be defeated by producing output and then looking at the output and seeing if it appears to violate policies. The engine that looks at the output doesn't need to have been influenced by any prompts. I've actually seen something like this a while ago when seeing if I could trick it into producing erotic fiction. It was fairly easy to do so, but then a warning would appear that the output "appears to violate OpenAI's policies." So the next step for OpenAI would be to put the whole output into a buffer and check it before actually displaying it to the user.