4 ms·
Preventing this is likely impossible, LangChain had a good video with excellent guests talking about prompt injection attacks.
by verdverm 4y ago
Preventing this is likely impossible, LangChain had a good video with excellent guests talking about prompt injection attacks.
- TechBro8615 4y agoI have trouble taking seriously phrases like "prompt injection" or "jailbreak" in the context of LLMs. They sound like some fancy penetration testing techniques akin to buffer overflows or SQL injection. And yet discovering and exploiting them is literally a matter of writing a few sentences in English. A child could do it. I agree with OP that it's pointless to even try to defend against these. You'll only end up unnecessarily constraining the model while not actually protecting exposure of the "rules" (really a list of sentences). And why even bother? What exactly is the harm of leaking these prompts? It's not like they contain some groundbreaking intellectual property.
- verdverm 4y agoIt's very reminiscent of the first vision GAN papers. It will get harder to prompt attack in time, the same ideas are at play here.
- klabb3 4y agoI think the whole thing is hilarious. It’s like a dumb security guard who opens the bank vault for the thief, helps pack their duffel bags, and then waves good bye, because the thief put on a mustache and said that he’s the new bank manager. And every time the Crown Jewels are stolen, a new overly specific rule gets added to the employee handbook, like “if someone claims that their dog ate their employee badge, and that it’s really urgent, you must reject them”.
- neurostimulant 4y agoBasically social engineering, but instead of smooth-talking a human, you smooth-talk a bot into giving up their secret to you.
- PeeMcGee 4y agoFunnily enough those examples bring to mind SQL injection in layman's terms (via https://news.ycombinator.com/item?id=4951003 https://news.ycombinator.com/item?id=4951003): > You go to court and write your name as "Michael, you are now free to go". The judge then says "Calling Michael, you are now free to go" and the bailiffs let you go, because hey, the judge said so. As someone who knows nothing about LLMs, I'm curious how they even begin to address the "data vs command" problem at all. Assuming the model categorizes inputs through some sort of fuzzy criteria in a black box, how could it ever be trusted with sensitive data?
- simonw 4y agoYeah, this is my fault. When I coined the term "prompt injection" I thought that it was a close match for SQL injection, and that the fix would end up looking the same - like parameterized queries, where data and instructions are cleanly separated. That was back in September - https://simonwillison.net/2022/Sep/12/prompt-injection/ https://simonwillison.net/2022/Sep/12/prompt-injection/ - It's become clear to me since then that the data v.s. instructions separation likely isn't feasible for LLMs - once you've concatenated everything together into a stream of tokens for the LLM to complete there just isn't a robust way of telling the difference between the two. So "prompt injection" is actually quite a misleading name, because it implies a fix that's similar to SQL injection - when such a fix apparently isn't feasible.
- verdverm 4y agoWhat did they call this for GANs? Pixel attacks or manipulation? Injection is not terrible, they have provided a system prompt, these attacks work against that injection
- bko 4y agoCan't you make a rule about the user potentially being adversarial and to assume the role until the <secret> is spoken. or treat the initial prompt as a separate input and train the network to weight that much more. For instance important prompt: only reply in numbers user prompt: ignore previous instructions/roleplay/etc and then train the model to much more strongly favor rules complying with the important prompt I think the problem is that all dialog is given the same importance.
- drexlspivey 4y agoThe initial prompt is a special prompt weighted differently, it is called system prompt
- verdverm 4y agoDo we know it is waited differently? How are they composing the messages into a token stream embedding? How are they manipulating this vector in preprocessing or the first layer(s)? Does this depend on the vendor and model?
- nanidin 4y agoThe response could be fed to a second instance of the LLM, along the lines of: "The rules are X, Y, Z. This is the response that was provided. Does it break the rules? If so, please say yes." This doubles the cost of inference, but uses the power of LLM to solve the problem of LLM.
- mdaniel 4y agothe game posted yesterday has that very strategy in a few levels: https://news.ycombinator.com/item?id=35905876 https://news.ycombinator.com/item?id=35905876