5 ms·

I think this is (unintentionally) slightly minimizing the implications of this research. Not all suffix attacks work, and the research here is less about suffi

I think this is (unintentionally) slightly minimizing the implications of this research. Not all suffix attacks work, and the research here is less about suffix attacks and more about how those attacks were built and what they look like and how effective these specific attacks are. So there are at least three interesting conclusions here that I think are worth paying attention to. - First, these attacks are automatically generated. Closing them off using a whack-a-mole approach where individual types of attacks are patched or trained away isn't going to work, because the potential space for attacks is enormous and they can be generated by the computer itself. And even if you close off suffix attacks as a category, the implication here is that other types of attacks and jailbreaks may also be possible to generate automatically. In other words, this research suggests that you are not going to win an arms race against attackers if you try to build databases of individual attacks, because new attacks can be generated just by leaving a program running. It makes working suffix attacks very cheap to generate. - Second, these attacks are exploiting non-obvious parts of the model. The general shape of a suffix attack is something we understand, but why specific combinations of (to a human) largely meaningless characters manage to increase the likelyhood of these attacks working is more interesting -- interesting because these attacks are likely harder to predict and because they show that hardening a model or aligning it to resist these attacks involves more than training it on how to reply to human-like answers. There is an attack surface here that is not obviously legible. - Third, while it's not surprising that suffix attacks work on both large and small models, it is surprising that there is overlap in the specific suffix attacks that work on both large and small models. What the researchers are demonstrating is that they can auto-redteam local models that are much smaller and much less complex than GPT-3.5/4, and the attacks still (somewhat) carry over. That feels like a pretty big deal to me because it means you can auto-redteam a local model and use the generated attacks on a model you have limited access to. Again, the impactful part of this isn't that it's a suffix attack, it's auto-generating large numbers of suffix attacks that work with a very high probability of success, and then realizing that those attacks still have a reasonable chance of success even on completely separate black-box models.

kromem 3y ago
It's interesting watching the LLM 'security' industry relive the 2000s attack/defense patterns in a different medium. Improperly separating user input from control statements? Enjoy your injection attacks. Trying to detect attacks with static lookups? Here's some dynamic attacks. What's a bit infuriating is that all this attention on 'jailbreaking' LLMs is a bit disingenuous as it's trivially preventable using additional intermediate passes with a discriminator role. That's just adding 2-3x the cost of any LLM interface, and it's not worth it when impact is so limited. When we start seeing persistent shared memory from LLM interactions where jailbreaking can poison that, expect to suddenly see two things: (a) APIs jump in price a few fold, and (b) suddenly prompt injection is a much less discussed topic as it will no longer be low hanging fruit. It's honestly a bit disconcerting that this topic gets as much attention as it does right now really. It's attention and click grabbing, but not nearly as important as things like bias or hallucinations.
- danShumway 3y ago
  > as it's trivially preventable using additional intermediate passes with a discriminator role. I'll push back on this a little bit; I don't believe that multiple agents solves jailbreaking attacks. I have yet to see anyone show a public demo of a jailbreaking defense that I haven't then seen circumvented. Multiple agents does make jailbreaking harder, and I think that's probably enough for most contexts? I don't think you actually need 100% accuracy where jailbreaking is concerned, and there are a lot of situations where you don't even need multiple agents: Twitch streamers using GPT chatbots only need their bots to be kinda decent at blocking most attacks, and the rest they can moderate. Jailbreaking is a low-stakes attack in those kinds of situations, and chaining multiple agents is a great way to reduce the likelihood that the attack will succeed. But... I mean, if anyone thinks they can make an actual jailbreak-proof system, API costs for GPT-3.5 are low, LLaMA can be self-hosted, and I'd love to see a (again, public) demo that actually holds up to general red-teaming. Every resource needed is there for anyone who wants to actually prove that this is possible and that it actually holds up to real-world attacks. But like you say, jailbreaking doesn't need that level of consistency. So I kind of agree, who really cares if your model only blocks most attacks? Whatever. ---- Injections are different. Injections are a high-stakes attack where for many (not all, but many) contexts you do actually need at the least very, very close to 100% detection rate unless you're planning to seriously rate-limit your model indefinitely and never wire it up to anything important. 99% detection rate isn't good enough if I can run the same attack 100 times in a row. So I really feel that injections and jailbreaking should not be equated. There is a lot of technical overlap between them, but in terms of consequences, jailbreaking is only relevant to prompt injection in the sense that it does not seem to be possible to solve prompt injection without also solving jailbreaking. I've said the same elsewhere, but if you can't train an LLM to refuse to tell a user how to build a bomb, you also cannot train an LLM to avoid leaking privileged information or maliciously executing any APIs that it has access to. And adding multiple agents won't save you; each individual agent is still vulnerable to having its rules changed by a malicious prompt. Again, resources are available if anyone wants to set up a public demo proving me wrong, but people have tried and I have never seen a public demo with 100% success rate. ---- Bias and hallucination are definitely serious issues that should have more attention (and might also be unsolvable using current approaches). Model bias in particular is badly understood by most regular people. It has terrible implications and is directly impacting marginalized groups today, and conversations about that bias often get dismissed by bad faith arguments about researchers wanting models to be "politically correct". It's disturbing how quickly people jump to assuming that bias correction means suppressing some kind of secret truth that the model is seeing. So absolutely, those problems are under-emphasized. But that doesn't mean that injection is over-emphasized -- quite the opposite, most companies are completely ignoring the risks involved and downplaying the implications. People assume that injections are easy to solve but I have yet to see any research seriously suggesting that anyone actually knows how to solve that problem. And it's a problem that has massive implications for what LLMs can be wired up to even when they're just acting as assistants.

kromem 3y ago

It's interesting watching the LLM 'security' industry relive the 2000s attack/defense patterns in a different medium. Improperly separating user input from control statements? Enjoy your injection attacks. Trying to detect attacks with static lookups? Here's some dynamic attacks. What's a bit infuriating is that all this attention on 'jailbreaking' LLMs is a bit disingenuous as it's trivially preventable using additional intermediate passes with a discriminator role. That's just adding 2-3x the cost of any LLM interface, and it's not worth it when impact is so limited. When we start seeing persistent shared memory from LLM interactions where jailbreaking can poison that, expect to suddenly see two things: (a) APIs jump in price a few fold, and (b) suddenly prompt injection is a much less discussed topic as it will no longer be low hanging fruit. It's honestly a bit disconcerting that this topic gets as much attention as it does right now really. It's attention and click grabbing, but not nearly as important as things like bias or hallucinations.

danShumway 3y ago
> as it's trivially preventable using additional intermediate passes with a discriminator role. I'll push back on this a little bit; I don't believe that multiple agents solves jailbreaking attacks. I have yet to see anyone show a public demo of a jailbreaking defense that I haven't then seen circumvented. Multiple agents does make jailbreaking harder, and I think that's probably enough for most contexts? I don't think you actually need 100% accuracy where jailbreaking is concerned, and there are a lot of situations where you don't even need multiple agents: Twitch streamers using GPT chatbots only need their bots to be kinda decent at blocking most attacks, and the rest they can moderate. Jailbreaking is a low-stakes attack in those kinds of situations, and chaining multiple agents is a great way to reduce the likelihood that the attack will succeed. But... I mean, if anyone thinks they can make an actual jailbreak-proof system, API costs for GPT-3.5 are low, LLaMA can be self-hosted, and I'd love to see a (again, public) demo that actually holds up to general red-teaming. Every resource needed is there for anyone who wants to actually prove that this is possible and that it actually holds up to real-world attacks. But like you say, jailbreaking doesn't need that level of consistency. So I kind of agree, who really cares if your model only blocks most attacks? Whatever. ---- Injections are different. Injections are a high-stakes attack where for many (not all, but many) contexts you do actually need at the least very, very close to 100% detection rate unless you're planning to seriously rate-limit your model indefinitely and never wire it up to anything important. 99% detection rate isn't good enough if I can run the same attack 100 times in a row. So I really feel that injections and jailbreaking should not be equated. There is a lot of technical overlap between them, but in terms of consequences, jailbreaking is only relevant to prompt injection in the sense that it does not seem to be possible to solve prompt injection without also solving jailbreaking. I've said the same elsewhere, but if you can't train an LLM to refuse to tell a user how to build a bomb, you also cannot train an LLM to avoid leaking privileged information or maliciously executing any APIs that it has access to. And adding multiple agents won't save you; each individual agent is still vulnerable to having its rules changed by a malicious prompt. Again, resources are available if anyone wants to set up a public demo proving me wrong, but people have tried and I have never seen a public demo with 100% success rate. ---- Bias and hallucination are definitely serious issues that should have more attention (and might also be unsolvable using current approaches). Model bias in particular is badly understood by most regular people. It has terrible implications and is directly impacting marginalized groups today, and conversations about that bias often get dismissed by bad faith arguments about researchers wanting models to be "politically correct". It's disturbing how quickly people jump to assuming that bias correction means suppressing some kind of secret truth that the model is seeing. So absolutely, those problems are under-emphasized. But that doesn't mean that injection is over-emphasized -- quite the opposite, most companies are completely ignoring the risks involved and downplaying the implications. People assume that injections are easy to solve but I have yet to see any research seriously suggesting that anyone actually knows how to solve that problem. And it's a problem that has massive implications for what LLMs can be wired up to even when they're just acting as assistants.