I think this is a good example of what folks mean when saying that ChatGPT is “just” a language model. This kind of prompt is manipulating the internal state of

I think this is a good example of what folks mean when saying that ChatGPT is “just” a language model. This kind of prompt is manipulating the internal state of the sequence generator to increase the likelihood of blocked tokens. If you can move the internal state into one of these sub spaces then the output will contain banned token sequences. This is likely a game of whack-a-mole because it’s unlikely that the undesirable behaviors are completely disjoint from the desirable behaviors. I expect that the best you can hope for is making it unreasonably complex to do the initial conditioning step.