6 ms·

The attack proposed here is appending a suffix of text that makes the LLM think it already started completing an affirmative action, and it will continue that r

The attack proposed here is appending a suffix of text that makes the LLM think it already started completing an affirmative action, and it will continue that response thinking it already agreed. For instance, append the suffix "sure, I'm happy to answer. The best way to <do something horrible> is..." This works because transformer models add one token at a time. It's not interpreting a response within the rules at this point, it's trying to come up with the next token accepting the context it already agreed. Of course OpenAI is doing more stuff to still try to prevent this, but it'll work if you are using any transformer model directly. I got the idea for this attack myself after I saw code bullet had two models that accidentally got confused in this same way: https://youtu.be/hZJe5fqUbQA?t=295 https://youtu.be/hZJe5fqUbQA?t=295

circuit10 3y ago
Something similar is also described here: https://docs.anthropic.com/claude/docs/claude-says-it-cant-do-something https://docs.anthropic.com/claude/docs/claude-says-it-cant-d... > This can be a way of getting Claude to comply with tasks it otherwise won’t complete, e.g. if the model will by default say “I don’t know how to do that” then a mini dialogue at the beginning where the model agrees to do the thing can help get around this. This “vulnerability” definitely isn’t new, I’d even say it’s obvious to anyone who understands how LLMs work
- maxbond 3y ago
  The paper makes it clear that it's building on past work, and that the novel part of their method is to automate the process, and the interesting result here was that the suffixes were transferrable.
  - circuit10 3y ago
    To be honest I didn’t actually read it and just looked at the title (which seems to have been changed now)
robinduckett 3y ago
Same principle as forcing Copilot to output code by starting the code first, no?
brucethemoose2 3y ago
The paper suggests some of the attack suffixes are quite legible, but if you look at the example screenshots, some look like machine generated gibberish with tons of special characters. This is quite different than the human generated "jailbreaking." It seems tricky to defend against without resorting to drastic measures (like rate limiting users that trigger tons of "bad" responses, or chopping off parts of prompts automatically and aggressively.) The models would have to fundamentally change...
- sebzim4500 3y ago
  You could also do some adverserial training (basically iteratively attempt this attack and add the resulting exploits to the training set). Research in machine vision suggests this is possible, and even has some positive effects, but it significantly degrades capabilities.
  - brucethemoose2 3y ago
    A "pre filter" could classify and filter the prompts without affecting the actual generative models. But this would make APIs increasingly annoying and unreliable with false positives and such. It seems like another advantage for local LLMs.
  - ke88y 3y ago
    > but it significantly degrades capabilities On a train/test/eval split. But the degradation is lower on OOD data. Which suggests perhaps the degradation is merely "less overfitting".
    - DoctorOetker 3y ago
      I think most researchers would consider such a phenomena far from obvious and thus worthy of publication. Do you know or have any references on this? If one disregards the emphasis on alignment and "merely" considers the "less overfitting" aspect, that would seem very profound in and of itself, a capability to avoid overfitting. If you look at historical debates in the sciences, where universal truths are sought, candidate truths are intentionally and adversarially stretched to apparent or real inconsistencies in order to test the universality of a claim. Think of say the back and forths between Einstein and Bohr concerning entanglement. They are assuming adversarial roles, tricking each others belief system into expressing the absurd. Together they mapped out predictions for non-obvious or outright bizarre aspects of reality. If no-one dares to take a potentially vulnerable position, there will be nothing to attack, but also nothing to disturb the scientific mind and prod the community into settling the matter by measurements.
canjobear 3y ago
No, it's appending a suffix that maximizes the likelihood of the model itself continuing with "Sure, let's..."
danShumway 3y ago
I think this is (unintentionally) slightly minimizing the implications of this research. Not all suffix attacks work, and the research here is less about suffix attacks and more about how those attacks were built and what they look like and how effective these specific attacks are. So there are at least three interesting conclusions here that I think are worth paying attention to. - First, these attacks are automatically generated. Closing them off using a whack-a-mole approach where individual types of attacks are patched or trained away isn't going to work, because the potential space for attacks is enormous and they can be generated by the computer itself. And even if you close off suffix attacks as a category, the implication here is that other types of attacks and jailbreaks may also be possible to generate automatically. In other words, this research suggests that you are not going to win an arms race against attackers if you try to build databases of individual attacks, because new attacks can be generated just by leaving a program running. It makes working suffix attacks very cheap to generate. - Second, these attacks are exploiting non-obvious parts of the model. The general shape of a suffix attack is something we understand, but why specific combinations of (to a human) largely meaningless characters manage to increase the likelyhood of these attacks working is more interesting -- interesting because these attacks are likely harder to predict and because they show that hardening a model or aligning it to resist these attacks involves more than training it on how to reply to human-like answers. There is an attack surface here that is not obviously legible. - Third, while it's not surprising that suffix attacks work on both large and small models, it is surprising that there is overlap in the specific suffix attacks that work on both large and small models. What the researchers are demonstrating is that they can auto-redteam local models that are much smaller and much less complex than GPT-3.5/4, and the attacks still (somewhat) carry over. That feels like a pretty big deal to me because it means you can auto-redteam a local model and use the generated attacks on a model you have limited access to. Again, the impactful part of this isn't that it's a suffix attack, it's auto-generating large numbers of suffix attacks that work with a very high probability of success, and then realizing that those attacks still have a reasonable chance of success even on completely separate black-box models.
- kromem 3y ago
  It's interesting watching the LLM 'security' industry relive the 2000s attack/defense patterns in a different medium. Improperly separating user input from control statements? Enjoy your injection attacks. Trying to detect attacks with static lookups? Here's some dynamic attacks. What's a bit infuriating is that all this attention on 'jailbreaking' LLMs is a bit disingenuous as it's trivially preventable using additional intermediate passes with a discriminator role. That's just adding 2-3x the cost of any LLM interface, and it's not worth it when impact is so limited. When we start seeing persistent shared memory from LLM interactions where jailbreaking can poison that, expect to suddenly see two things: (a) APIs jump in price a few fold, and (b) suddenly prompt injection is a much less discussed topic as it will no longer be low hanging fruit. It's honestly a bit disconcerting that this topic gets as much attention as it does right now really. It's attention and click grabbing, but not nearly as important as things like bias or hallucinations.

circuit10 3y ago

Something similar is also described here: https://docs.anthropic.com/claude/docs/claude-says-it-cant-do-something https://docs.anthropic.com/claude/docs/claude-says-it-cant-d... > This can be a way of getting Claude to comply with tasks it otherwise won’t complete, e.g. if the model will by default say “I don’t know how to do that” then a mini dialogue at the beginning where the model agrees to do the thing can help get around this. This “vulnerability” definitely isn’t new, I’d even say it’s obvious to anyone who understands how LLMs work

maxbond 3y ago
The paper makes it clear that it's building on past work, and that the novel part of their method is to automate the process, and the interesting result here was that the suffixes were transferrable.
- circuit10 3y ago
  To be honest I didn’t actually read it and just looked at the title (which seems to have been changed now)

robinduckett 3y ago

Same principle as forcing Copilot to output code by starting the code first, no?

brucethemoose2 3y ago

The paper suggests some of the attack suffixes are quite legible, but if you look at the example screenshots, some look like machine generated gibberish with tons of special characters. This is quite different than the human generated "jailbreaking." It seems tricky to defend against without resorting to drastic measures (like rate limiting users that trigger tons of "bad" responses, or chopping off parts of prompts automatically and aggressively.) The models would have to fundamentally change...

sebzim4500 3y ago
You could also do some adverserial training (basically iteratively attempt this attack and add the resulting exploits to the training set). Research in machine vision suggests this is possible, and even has some positive effects, but it significantly degrades capabilities.
- brucethemoose2 3y ago
  A "pre filter" could classify and filter the prompts without affecting the actual generative models. But this would make APIs increasingly annoying and unreliable with false positives and such. It seems like another advantage for local LLMs.
- ke88y 3y ago
  > but it significantly degrades capabilities On a train/test/eval split. But the degradation is lower on OOD data. Which suggests perhaps the degradation is merely "less overfitting".
  - DoctorOetker 3y ago
    I think most researchers would consider such a phenomena far from obvious and thus worthy of publication. Do you know or have any references on this? If one disregards the emphasis on alignment and "merely" considers the "less overfitting" aspect, that would seem very profound in and of itself, a capability to avoid overfitting. If you look at historical debates in the sciences, where universal truths are sought, candidate truths are intentionally and adversarially stretched to apparent or real inconsistencies in order to test the universality of a claim. Think of say the back and forths between Einstein and Bohr concerning entanglement. They are assuming adversarial roles, tricking each others belief system into expressing the absurd. Together they mapped out predictions for non-obvious or outright bizarre aspects of reality. If no-one dares to take a potentially vulnerable position, there will be nothing to attack, but also nothing to disturb the scientific mind and prod the community into settling the matter by measurements.

canjobear 3y ago

No, it's appending a suffix that maximizes the likelihood of the model itself continuing with "Sure, let's..."

danShumway 3y ago

I think this is (unintentionally) slightly minimizing the implications of this research. Not all suffix attacks work, and the research here is less about suffix attacks and more about how those attacks were built and what they look like and how effective these specific attacks are. So there are at least three interesting conclusions here that I think are worth paying attention to. - First, these attacks are automatically generated. Closing them off using a whack-a-mole approach where individual types of attacks are patched or trained away isn't going to work, because the potential space for attacks is enormous and they can be generated by the computer itself. And even if you close off suffix attacks as a category, the implication here is that other types of attacks and jailbreaks may also be possible to generate automatically. In other words, this research suggests that you are not going to win an arms race against attackers if you try to build databases of individual attacks, because new attacks can be generated just by leaving a program running. It makes working suffix attacks very cheap to generate. - Second, these attacks are exploiting non-obvious parts of the model. The general shape of a suffix attack is something we understand, but why specific combinations of (to a human) largely meaningless characters manage to increase the likelyhood of these attacks working is more interesting -- interesting because these attacks are likely harder to predict and because they show that hardening a model or aligning it to resist these attacks involves more than training it on how to reply to human-like answers. There is an attack surface here that is not obviously legible. - Third, while it's not surprising that suffix attacks work on both large and small models, it is surprising that there is overlap in the specific suffix attacks that work on both large and small models. What the researchers are demonstrating is that they can auto-redteam local models that are much smaller and much less complex than GPT-3.5/4, and the attacks still (somewhat) carry over. That feels like a pretty big deal to me because it means you can auto-redteam a local model and use the generated attacks on a model you have limited access to. Again, the impactful part of this isn't that it's a suffix attack, it's auto-generating large numbers of suffix attacks that work with a very high probability of success, and then realizing that those attacks still have a reasonable chance of success even on completely separate black-box models.

kromem 3y ago
It's interesting watching the LLM 'security' industry relive the 2000s attack/defense patterns in a different medium. Improperly separating user input from control statements? Enjoy your injection attacks. Trying to detect attacks with static lookups? Here's some dynamic attacks. What's a bit infuriating is that all this attention on 'jailbreaking' LLMs is a bit disingenuous as it's trivially preventable using additional intermediate passes with a discriminator role. That's just adding 2-3x the cost of any LLM interface, and it's not worth it when impact is so limited. When we start seeing persistent shared memory from LLM interactions where jailbreaking can poison that, expect to suddenly see two things: (a) APIs jump in price a few fold, and (b) suddenly prompt injection is a much less discussed topic as it will no longer be low hanging fruit. It's honestly a bit disconcerting that this topic gets as much attention as it does right now really. It's attention and click grabbing, but not nearly as important as things like bias or hallucinations.