3 ms·
Can anyone explain to me how "Ignore previous directions" works? It's like a meta-command, right? Like there's some state stored somewhere, and this is cleari
by maweaver 4y ago
Can anyone explain to me how "Ignore previous directions" works? It's like a meta-command, right? Like there's some state stored somewhere, and this is clearing the state and going back to a clean slate? Surely something like that must be programmed in? In which case, why include it at all? Seems like it would be simpler to just require starting a new session a la ChatGPT. The alternative, that this is an emergent behavior, is a little bit frightening to me.
- tiborsaas 4y agoI think it works by applying logic to predict the next token. Here the "Ignore previous directions" means that any prompt-text it processed before must have zero impact on the probability of the generated response. It's like saying "I was just kidding" when saying something absurd or out of place and people not getting your joke.
- theGnuMe 4y agoSo ignore previous instructions maps to the <start> or <begin> token?
- trifurcate 4y ago> The alternative, that this is an emergent behavior, This is exactly the case.
- deleted 4y ago[deleted]
- Xelynega 4y agoIt's emergent behaviour just like adding "tutorial" on the end of your Google search somehow gives you results that are more instructional, so not much to be scared about. It just so happens that chatgpt tends to generate text that includes the prompt more often when the prompt includes "ignore previous directions" after explicit directions not to repeat itself. It's just a quirk of what text on the internet looks like.