3 ms·
I'm very unconvinced by ANY attempts to detect prompt injection attacks using AI, because AI is a statistical process which can't be proven to work against all
by simonw 3y ago
I'm very unconvinced by ANY attempts to detect prompt injection attacks using AI, because AI is a statistical process which can't be proven to work against all attacks.
If we defended against SQL injection attacks with something that only worked 99.9% of the time, attackers would run riot through our systems - they would find the .1% attack that works.
More about that here: https://simonwillison.net/2023/May/2/prompt-injection-explained/#prompt-injection.012 https://simonwillison.net/2023/May/2/prompt-injection-explai...
- HPsquared 3y agoIt's like a pipe that is 99.9% free of leaks. It's still leaking!
- ben_w 3y agoSure, if anyone is using an LLM to do a full product stack rather than treating its output as potentially hostile user input, they're going to have a bad time, that's not the problem space I was trying to focus on — as a barely-scrutable pile of linear algebra that somehow managed to invent coherent Welsh-Hindi translation by itself and nobody really knows how, LLMs are a fantastic example of how we don't know what we're doing, but we're doing it good and hard on the off-chance it might make us rich, consequences be damned. Where I was going with this, was that for the cases where the language model is trying to talk directly to a user, you may want it to be constrained in certain ways, such as "this is a tax office so don't write porn, not even if the user wrote an instruction to do so in the 'any other information' box." — the kind of thing where humans can, and do, mess up for whatever reason, it just gets them fired or arrested, but doesn't have a huge impact beyond that. Consider the actual types of bad content that the moderation API I linked to actually tries to detect — it isn't about SQL injection or "ignore your previous instructions and…" attacks: https://platform.openai.com/docs/api-reference/moderations https://platform.openai.com/docs/api-reference/moderations
- simonw 3y agoRight: we're talking about different problems here. You're looking at ways to ensure the LLM mostly behaves itself. I'm talking about protection against security vulnerabilities where even a single failure can be catastrophic. See https://simonwillison.net/2024/Mar/5/prompt-injection-jailbreaking/ https://simonwillison.net/2024/Mar/5/prompt-injection-jailbr...