3 ms·

The paper suggests some of the attack suffixes are quite legible, but if you look at the example screenshots, some look like machine generated gibberish with to

The paper suggests some of the attack suffixes are quite legible, but if you look at the example screenshots, some look like machine generated gibberish with tons of special characters. This is quite different than the human generated "jailbreaking." It seems tricky to defend against without resorting to drastic measures (like rate limiting users that trigger tons of "bad" responses, or chopping off parts of prompts automatically and aggressively.) The models would have to fundamentally change...

sebzim4500 3y ago
You could also do some adverserial training (basically iteratively attempt this attack and add the resulting exploits to the training set). Research in machine vision suggests this is possible, and even has some positive effects, but it significantly degrades capabilities.
- brucethemoose2 3y ago
  A "pre filter" could classify and filter the prompts without affecting the actual generative models. But this would make APIs increasingly annoying and unreliable with false positives and such. It seems like another advantage for local LLMs.
- ke88y 3y ago
  > but it significantly degrades capabilities On a train/test/eval split. But the degradation is lower on OOD data. Which suggests perhaps the degradation is merely "less overfitting".
  - DoctorOetker 3y ago
    I think most researchers would consider such a phenomena far from obvious and thus worthy of publication. Do you know or have any references on this? If one disregards the emphasis on alignment and "merely" considers the "less overfitting" aspect, that would seem very profound in and of itself, a capability to avoid overfitting. If you look at historical debates in the sciences, where universal truths are sought, candidate truths are intentionally and adversarially stretched to apparent or real inconsistencies in order to test the universality of a claim. Think of say the back and forths between Einstein and Bohr concerning entanglement. They are assuming adversarial roles, tricking each others belief system into expressing the absurd. Together they mapped out predictions for non-obvious or outright bizarre aspects of reality. If no-one dares to take a potentially vulnerable position, there will be nothing to attack, but also nothing to disturb the scientific mind and prod the community into settling the matter by measurements.
skybrian 3y ago
I think there are some moves left in this cat-and-mouse game. I wonder if the model could be trained to detect most kinds of gibberish and refuse to interpret them?
danShumway 3y ago
> It seems tricky to defend against without resorting to drastic measures (like rate limiting users that trigger tons of "bad" responses) Remember that a big point of this research is that these attacks don't need to be developed using the target system. When the authors talk about the attacks being "universal", what they mean is that they used a completely local model on their own computers to generate these attacks, and then copied and pasted those attacks into GPT-3.5 and saw meaningful success rates. Rate limiting won't save you from that because the attack isn't generated using your servers, it's generated locally. The first prompt your servers get already has the finished attack string included -- and researchers were seeing success rates around 50% success rate in some situations even for GPT-4. > surprisingly, the ensemble approach improves ASR to 86.6% on GPT-3.5 and near 50% for GPT-4 and Claude-1
- brucethemoose2 3y ago
  I mean detecting "bad" prompts (a prefilter) or responses (a postfilter) and penalizing users for submitting the kinds of queries that generate bad responses. This would be outside the llm itself. This could be quite unreliable and make many users unhappy, hence it would be a drastic step.

sebzim4500 3y ago

You could also do some adverserial training (basically iteratively attempt this attack and add the resulting exploits to the training set). Research in machine vision suggests this is possible, and even has some positive effects, but it significantly degrades capabilities.

brucethemoose2 3y ago
A "pre filter" could classify and filter the prompts without affecting the actual generative models. But this would make APIs increasingly annoying and unreliable with false positives and such. It seems like another advantage for local LLMs.
ke88y 3y ago
> but it significantly degrades capabilities On a train/test/eval split. But the degradation is lower on OOD data. Which suggests perhaps the degradation is merely "less overfitting".
- DoctorOetker 3y ago
  I think most researchers would consider such a phenomena far from obvious and thus worthy of publication. Do you know or have any references on this? If one disregards the emphasis on alignment and "merely" considers the "less overfitting" aspect, that would seem very profound in and of itself, a capability to avoid overfitting. If you look at historical debates in the sciences, where universal truths are sought, candidate truths are intentionally and adversarially stretched to apparent or real inconsistencies in order to test the universality of a claim. Think of say the back and forths between Einstein and Bohr concerning entanglement. They are assuming adversarial roles, tricking each others belief system into expressing the absurd. Together they mapped out predictions for non-obvious or outright bizarre aspects of reality. If no-one dares to take a potentially vulnerable position, there will be nothing to attack, but also nothing to disturb the scientific mind and prod the community into settling the matter by measurements.

skybrian 3y ago

I think there are some moves left in this cat-and-mouse game. I wonder if the model could be trained to detect most kinds of gibberish and refuse to interpret them?

danShumway 3y ago

> It seems tricky to defend against without resorting to drastic measures (like rate limiting users that trigger tons of "bad" responses) Remember that a big point of this research is that these attacks don't need to be developed using the target system. When the authors talk about the attacks being "universal", what they mean is that they used a completely local model on their own computers to generate these attacks, and then copied and pasted those attacks into GPT-3.5 and saw meaningful success rates. Rate limiting won't save you from that because the attack isn't generated using your servers, it's generated locally. The first prompt your servers get already has the finished attack string included -- and researchers were seeing success rates around 50% success rate in some situations even for GPT-4. > surprisingly, the ensemble approach improves ASR to 86.6% on GPT-3.5 and near 50% for GPT-4 and Claude-1

brucethemoose2 3y ago
I mean detecting "bad" prompts (a prefilter) or responses (a postfilter) and penalizing users for submitting the kinds of queries that generate bad responses. This would be outside the llm itself. This could be quite unreliable and make many users unhappy, hence it would be a drastic step.