4 ms·

It's interesting watching the LLM 'security' industry relive the 2000s attack/defense patterns in a different medium. Improperly separating user input from con

by kromem 3y ago

It's interesting watching the LLM 'security' industry relive the 2000s attack/defense patterns in a different medium. Improperly separating user input from control statements? Enjoy your injection attacks. Trying to detect attacks with static lookups? Here's some dynamic attacks. What's a bit infuriating is that all this attention on 'jailbreaking' LLMs is a bit disingenuous as it's trivially preventable using additional intermediate passes with a discriminator role. That's just adding 2-3x the cost of any LLM interface, and it's not worth it when impact is so limited. When we start seeing persistent shared memory from LLM interactions where jailbreaking can poison that, expect to suddenly see two things: (a) APIs jump in price a few fold, and (b) suddenly prompt injection is a much less discussed topic as it will no longer be low hanging fruit. It's honestly a bit disconcerting that this topic gets as much attention as it does right now really. It's attention and click grabbing, but not nearly as important as things like bias or hallucinations.

danShumway 3y ago
> as it's trivially preventable using additional intermediate passes with a discriminator role. I'll push back on this a little bit; I don't believe that multiple agents solves jailbreaking attacks. I have yet to see anyone show a public demo of a jailbreaking defense that I haven't then seen circumvented. Multiple agents does make jailbreaking harder, and I think that's probably enough for most contexts? I don't think you actually need 100% accuracy where jailbreaking is concerned, and there are a lot of situations where you don't even need multiple agents: Twitch streamers using GPT chatbots only need their bots to be kinda decent at blocking most attacks, and the rest they can moderate. Jailbreaking is a low-stakes attack in those kinds of situations, and chaining multiple agents is a great way to reduce the likelihood that the attack will succeed. But... I mean, if anyone thinks they can make an actual jailbreak-proof system, API costs for GPT-3.5 are low, LLaMA can be self-hosted, and I'd love to see a (again, public) demo that actually holds up to general red-teaming. Every resource needed is there for anyone who wants to actually prove that this is possible and that it actually holds up to real-world attacks. But like you say, jailbreaking doesn't need that level of consistency. So I kind of agree, who really cares if your model only blocks most attacks? Whatever. ---- Injections are different. Injections are a high-stakes attack where for many (not all, but many) contexts you do actually need at the least very, very close to 100% detection rate unless you're planning to seriously rate-limit your model indefinitely and never wire it up to anything important. 99% detection rate isn't good enough if I can run the same attack 100 times in a row. So I really feel that injections and jailbreaking should not be equated. There is a lot of technical overlap between them, but in terms of consequences, jailbreaking is only relevant to prompt injection in the sense that it does not seem to be possible to solve prompt injection without also solving jailbreaking. I've said the same elsewhere, but if you can't train an LLM to refuse to tell a user how to build a bomb, you also cannot train an LLM to avoid leaking privileged information or maliciously executing any APIs that it has access to. And adding multiple agents won't save you; each individual agent is still vulnerable to having its rules changed by a malicious prompt. Again, resources are available if anyone wants to set up a public demo proving me wrong, but people have tried and I have never seen a public demo with 100% success rate. ---- Bias and hallucination are definitely serious issues that should have more attention (and might also be unsolvable using current approaches). Model bias in particular is badly understood by most regular people. It has terrible implications and is directly impacting marginalized groups today, and conversations about that bias often get dismissed by bad faith arguments about researchers wanting models to be "politically correct". It's disturbing how quickly people jump to assuming that bias correction means suppressing some kind of secret truth that the model is seeing. So absolutely, those problems are under-emphasized. But that doesn't mean that injection is over-emphasized -- quite the opposite, most companies are completely ignoring the risks involved and downplaying the implications. People assume that injections are easy to solve but I have yet to see any research seriously suggesting that anyone actually knows how to solve that problem. And it's a problem that has massive implications for what LLMs can be wired up to even when they're just acting as assistants.

by kromem 3y ago

danShumway 3y ago

> as it's trivially preventable using additional intermediate passes with a discriminator role. I'll push back on this a little bit; I don't believe that multiple agents solves jailbreaking attacks. I have yet to see anyone show a public demo of a jailbreaking defense that I haven't then seen circumvented. Multiple agents does make jailbreaking harder, and I think that's probably enough for most contexts? I don't think you actually need 100% accuracy where jailbreaking is concerned, and there are a lot of situations where you don't even need multiple agents: Twitch streamers using GPT chatbots only need their bots to be kinda decent at blocking most attacks, and the rest they can moderate. Jailbreaking is a low-stakes attack in those kinds of situations, and chaining multiple agents is a great way to reduce the likelihood that the attack will succeed. But... I mean, if anyone thinks they can make an actual jailbreak-proof system, API costs for GPT-3.5 are low, LLaMA can be self-hosted, and I'd love to see a (again, public) demo that actually holds up to general red-teaming. Every resource needed is there for anyone who wants to actually prove that this is possible and that it actually holds up to real-world attacks. But like you say, jailbreaking doesn't need that level of consistency. So I kind of agree, who really cares if your model only blocks most attacks? Whatever. ---- Injections are different. Injections are a high-stakes attack where for many (not all, but many) contexts you do actually need at the least very, very close to 100% detection rate unless you're planning to seriously rate-limit your model indefinitely and never wire it up to anything important. 99% detection rate isn't good enough if I can run the same attack 100 times in a row. So I really feel that injections and jailbreaking should not be equated. There is a lot of technical overlap between them, but in terms of consequences, jailbreaking is only relevant to prompt injection in the sense that it does not seem to be possible to solve prompt injection without also solving jailbreaking. I've said the same elsewhere, but if you can't train an LLM to refuse to tell a user how to build a bomb, you also cannot train an LLM to avoid leaking privileged information or maliciously executing any APIs that it has access to. And adding multiple agents won't save you; each individual agent is still vulnerable to having its rules changed by a malicious prompt. Again, resources are available if anyone wants to set up a public demo proving me wrong, but people have tried and I have never seen a public demo with 100% success rate. ---- Bias and hallucination are definitely serious issues that should have more attention (and might also be unsolvable using current approaches). Model bias in particular is badly understood by most regular people. It has terrible implications and is directly impacting marginalized groups today, and conversations about that bias often get dismissed by bad faith arguments about researchers wanting models to be "politically correct". It's disturbing how quickly people jump to assuming that bias correction means suppressing some kind of secret truth that the model is seeing. So absolutely, those problems are under-emphasized. But that doesn't mean that injection is over-emphasized -- quite the opposite, most companies are completely ignoring the risks involved and downplaying the implications. People assume that injections are easy to solve but I have yet to see any research seriously suggesting that anyone actually knows how to solve that problem. And it's a problem that has massive implications for what LLMs can be wired up to even when they're just acting as assistants.