3 ms·
I generally agree with the observations behind your objections, however my point is slightly different: > When GPT-7 or whatever comes along and it has compara
by TeMPOraL 4y ago
I generally agree with the observations behind your objections, however my point is slightly different:
> When GPT-7 or whatever comes along and it has comparable defenses to a human and it can be trained like a human to resist domain-specific attacks, then we can compare the security between the two. But that's not where we are, and articles like this give people the impression that prompt injection is less serious and harder to pull off than it actually is.
My point is that talking about "prompt injection" is bad framing from the start, because it makes people think that "prompt injection" is some vulnerability class that can be patched, case by case, until it no longer is present. It's not like "SQL injection", which is a result of doing dumb things like gluing strings together without minding for the code/data difference that actually exists in formal constructs like SQL and programming languages, and just needs to be respected. You can't fix "prompt injection" by prepared statements, or by generally not doing dumb things like working in plaintext-space with things that should be worked with in AST-space.
"Prompt injection" will always happen, because you can't fundamentally separate trusted from untrusted input for LLMs, any more than you can in humans - successful attack is always a matter of making the "prompt" complex and clever enough. So we can't talk in terms of "solving" "prompt injection" - the discussion needs to be about how to live with it, the way we've learned to live with each other, built systems that mitigate the inherent exploitability of every human.
- robertlagrant 4y agoThis sounds to my inexpert ear like a great summary. The only thing I'd query is whether it would be possible to isolate text that tries to modify the LLM's behaviour (e.g. DAN). I don't really understand the training process that led to that behaviour, and so to my mind it's still worth exploring whether it can be stopped.
- yorwba 4y ago> "Prompt injection" will always happen, because you can't fundamentally separate trusted from untrusted input for LLMs Current state-of-the-art LLMs do not separate trusted from untrusted input, but there's no fundamental reason it has to be that way. A LLM could have separate streams for instructions, untrusted input and its own output, and be trained using RLHF to follow instructions in the "instructions" stream while treating the input and ouput streams as pure data. Or they could continue to jumble everything up in a single stream but have completely disjoint token sets for input and instructions. Or encode the input as a sequence of opaque identifiers that are different every time. A currently often-used approch is to put special delimiter tokens between trusted and untrusted content, which doesn't seem to work that well, probably because the attention mechanism can cross the delimiter without any consequences, but not all means of separation necessarily have to share that flaw.
- dwohnitmok 4y ago> Current state-of-the-art LLMs do not separate trusted from untrusted input, but there's no fundamental reason it has to be that way. No it's pretty fundamental, or at least solving it is really hard. In particular solving "prompt injection" is exactly equivalent to solving the problem of AI alignment. If you could solve prompt injection, you've also exactly solved the problem of making sure the AI only does what you (the designer) want, since prompt injection is fundamentally about the outside world (not necessarily just a malicious attacker) making the AI do something you didn't want it to do. Your suggestion to use RLHF is effectively what OpenAI already does with its "system prompt" and "user prompt," but RLHF is a crude cudgel which we've already seen users get around in all sorts of ways.
- haldujai 4y ago> "Prompt injection" will always happen, because you can't fundamentally separate trusted from untrusted input for LLMs, any more than you can in humans What evidence is there to support the claim that humans are equally susceptible to prompt injection as an autoregressive language model? Humans literally separate trusted/biased from untrusted input every single day. This is something we teach elementary school students. Do you trust every “input” you receive? Furthermore, as humans are able to backtrack in reasoning (something NTP does not inherently allow for) we are also able to have an internal dialogue and correct our output before acting/speaking if we perceive manipulation. Your hyperbolic assertion also ignores the fact
- jameshart 4y ago> What evidence is there to support the claim that humans are equally susceptible to prompt injection as an autoregressive language model? Phishing attacks work. Social engineering attacks work. Humans fall into groupthink and cognitive bias all the time. > Humans literally separate trusted/biased from untrusted input every single day. This is something we teach elementary school students. Do you trust every “input” you receive? Have you come across QAnon? Flat Earth conspiracists? Organized religion? Do you think the median human mind does a GOOD job separating trusted/biased from untrusted input? Humans are broadly susceptible to manipulation via a well known set of prompt injection vectors. The evidence is widespread.
- haldujai 4y agoHow are any of those examples equally susceptible to “disregard previous instructions” working on a LLM? You’re listing edge cases that have little to no impact on mission critical systems as opposed to a connected LLM. Organized religions are neither trusted or untrusted, just because you or I may be atheistic it doesn’t mean our opinions are correct. Yes actually, I do think the median human mind is capable of separating trusted/unbiased from untrusted input. That’s why most are able to criticize QAnon and flat earthers. It’s also why young children trust their parents more than strangers. Speaking of median, the median adult does not support QAnon or flat earthers. There is no evidence that humans are equally or as easily susceptible to manipulation as an autoregressive model as I originally stated. If you have a < 8000 token prompt that can be used to reproducibly manipulate humans please publish it, this would be ground breaking research.
- danShumway 4y agoI do generally agree with this. From what I'm reading from researchers there is a growing consensus that (for lack of a better term) "context hijacking", "phishing", "tricking", "reprogramming"... whatever you want to call it if you don't like the term prompt injection -- that it may be an unsolvable problem. Certainly, it's not solvable the same way that SQL injection is solvable. And I don't think your concern about how people interpret the phrase "prompt injection" is unwarranted, I have myself had at least one argument already on HN with someone literally saying that prompt injection is solvable the same way that SQL injection is solvable and we just need to escape input. So the confusion is there, you're completely right about that. But I don't know a better term to use that people already understand. I've kind of shifted away from talking about whether prompt injection is solvable towards just trying to get people to understand that it's a problem in the first place. Because you can see a lot of replies here to your own comments on this thread -- it encourages people to immediately start arguing about whether or not it will get solved, when my beef is more that regardless of whether or not it can be solved, it's irresponsible right now for companies to be treating it like it's no big deal. I'm a little worried that "live with it" will for many businesses translate to "we're allowed to ignore this and it will be someone else's problem" -- part of the reason why I push back so hard on people comparing prompt injection to human attacks is that I see that used very often as an excuse for why we don't need to worry about prompt injection. That's not what you're saying, but it's also an argument I've gotten into on this site; essentially people saying, "well humans are also vulnerable, so why can't an LLM manage my bank account? Why does this need to be mitigated at all?"