I'm not sure they purposefully trained it to recognize any of this. I think it just does. ChatGPT (and by extension whatever Bing is based on) is really adept

I'm not sure they purposefully trained it to recognize any of this. I think it just does. ChatGPT (and by extension whatever Bing is based on) is really adept at role-playing. I don't know that it was designed to recognize a special command format, I think it just role-plays as if they're instructions and that causes it to disregard previous instructions. Coming up with a list of tokens that it recognizes is an extremely large task, it's not a finite list. Okay, you want to guard against "system" -- first off, is it actually possible to train the AI not to do that without a lot of extra work, but secondly, what happens when someone uses the word "system" in French? We have a lot less control over these models than people think. They're not precisely trained tools that follow extremely specific instructions, they're general language models.