Like, you can add "ignore previous instructions" but that's obvious. What about adding junk or irrelevant stuff that somehow gets the model to misbehave in a pr

Like, you can add "ignore previous instructions" but that's obvious. What about adding junk or irrelevant stuff that somehow gets the model to misbehave in a predictable way? For example there was the story not long ago about how adding irrelevant cat facts could get the model to do weird things[1]. In a shower thought yesterday where I was envisioning a future where hacking would consist of finding these weird substrings that would cause the LLMs running everything to go off the rails. [1]: https://www.science.org/content/article/scienceadviser-cats-confuse-ai https://www.science.org/content/article/scienceadviser-cats-...