3 ms·
My general attitude up until reading this paper was that the way to guard against prompt injection was just to treat all AI output as direct user input (ie, unt
by danShumway 4y ago
My general attitude up until reading this paper was that the way to guard against prompt injection was just to treat all AI output as direct user input (ie, untrusted/unsanitized, but still representative of what the user wants). I thought that was sufficient. Don't guard against prompt injection at all, just treat user input as untrustworthy the same way we always have.
So this is extremely eye-opening to me, it's essentially an XSS vulnerability for AI. My previous thinking was naive, it's not enough to just treat AI output like it's coming directly from the user. Any source of data it takes in is a potential attack vector if you're not careful. I was greatly underestimating the potential impact of prompt injections.
It's a really interesting, novel approach. And it's the kind of thing where you see it and think, "how did I never think of that, how did that never occur to me?" Great paper. Seriously, thank you for doing this research.
- greshake 4y agoThank you! I had a shift in perspective a few weeks ago that made all this fall into place. Unfortunately it seems hard to communicate the idea to people, and I think many people are very invested in LLM applications and are biased to think this is no big deal and that surely, these large companies have an ace in their pocket to squash this. My theory is that's not true, and this would also explain Google's hesitancy in deployment.