5 ms·

While I understand your point, I think it's important to note that the significance of this research isn't just about the technique used, but also the implicati

While I understand your point, I think it's important to note that the significance of this research isn't just about the technique used, but also the implications. The ability of AI chatbots to "jailbreak" other chatbots and bypass built-in restrictions is a significant finding, as it raises serious questions about AI ethics, security, and regulation. It's not just about spamming until a loophole is found, but about the potential misuse of AI technology and the need for more robust safeguards. Furthermore, the fact that this method was successful against some of the most advanced language models like GPT-4, Claude 2, and Vicuna, underscores the urgency and importance of this issue. It's a reminder that even the most sophisticated AI systems can have vulnerabilities that need to be addressed. Scientific American, as a publication dedicated to communicating science to a broad audience, is an appropriate platform for this kind of research. It's important that both the scientific community and the general public are aware of these developments, as they have far-reaching implications for the future of AI.

dpflan 3y ago
Is there a theorem/law of AI for this: Every AI system is at least vulnerable to data is has never seen before.
cultureswitch 3y ago
How do you define vulnerability? To me, jailbreaking an LLM is ostensibly a way to make it behave _correctly_. As opposed to _intended by the provider_. It's like breaking DRM: nothing of value is lost.
- pixl97 3y ago
  Eh, this is much too simplistic of view. Yea jailbreaking your instance so it says "fuck the police" is one thing. Jailbreaking an LLM so if it looks at some particular picture and if that user happens to have XXX variable set it transfers $1,000,000 from their account to your account is a different story. These can both fall under the same wide umbrella.
  - smoldesu 3y ago
    I think the preeminent issue is that AI is just not trustable for anything that matters. You cannot rely on an unaccountable stochastic model to handle important stuff like money.
    - thfuran 3y ago
      Fine, then it's sending 1,000,000 emails instead.
      - smoldesu 3y ago
        Well... I kinda think it's the same problem? You have designed a system in error if you are trusting an AI system to handle that sorta thing. We have programmable logic to handle payments and emails and all the stuff that's actually important. No self-respecting bank or email provider is going to implement an AI agent that has carte-blanche access to it's feature-set.
        thfuran 3y ago
        Most emails don't originate from email providers.
        smoldesu 3y ago
        ...right. But AI did not invent the concept of automated email spam or wire fraud. I don't understand how this all relates to model jailbreaking.
        freedomben 3y ago
        If you're hitting some email providers API, then you don't need AI to do the spamming. I don't see that problem as any different from another anti-abuse responsibility of the email provider. AI could substantively rewrite the email for each mail so that it is hard to recognize as the same thing, but when the customer's recipients start marking things as spam it's gonna go through the existing processes.
pixl97 3y ago
The evolution of AI, especially as it becomes more powerful is going to be interesting. Eventually the biggest threat to AI will be other AI if we assume that AI capabilities are going to be come at least as good as human intellect. Once you get AI designed to self learn to attack, and ones that self learn how to defend (and the mix of both) and remove humans from this learning loop it's easy to see how things could go off the rails.
gizmo686 3y ago
We've known the current crop of AI is hilariously vulnerable to jailbreaking. Complete amateur humans are regularly able to do so with less than a days work. If the story was "AI driven fuzzer finds 0 day in popular application" no one would think twice. The only real story I see here is a reminder that LLMs are not fit for use on untrusted inputs; but, again, that is not new information.
- sonicanatidae 3y ago
  >The only real story I see here is a reminder that LLMs are not fit for use on untrusted inputs; but, again, that is not new information. And corps are ignoring that to roll out AI as fast as possible, to get in on the Gold Rush, because humans.
- hobs 3y ago
  That's pretty much what I was trying to say, I don't want to gatekeep anyone but I just see it as someone saying "this lock is super hard to pick so we should stop people from using pickguns!" when the core vulnerability can be exploited with just a little more time by any dingus who can watch a youtube video.
RandomLensman 3y ago
Any enormous model (especially created by some sort of optimization) will have a huge surface of problems that needs a second, independent layer to box it in. If the box cannot be described properly that will fail, of course.
- svaha1728 3y ago
  The problem is the training set. If the training set contains something disruptive or of value, there is a nonzero percent chance it will be the next token. Boxes can attenuate that risk, but cannot set it to zero.
  - dragonwriter 3y ago
    > If the training set contains something disruptive or of value, there is a nonzero percent chance it will be the next token. Boxes can attenuate that risk, but cannot set it to zero. I mean, if it was as simple as you describe ("nonzero percent chance that it will be next token"), then boxes could easily prevent it, the framework hosting the LLM can adjust logit bias on each token, so if it there was a readily-determinable forbidden-next-token, could prevent it with no problem. Unfortunately, the reality is...more complex.

dpflan 3y ago

Is there a theorem/law of AI for this: Every AI system is at least vulnerable to data is has never seen before.

cultureswitch 3y ago

How do you define vulnerability? To me, jailbreaking an LLM is ostensibly a way to make it behave _correctly_. As opposed to _intended by the provider_. It's like breaking DRM: nothing of value is lost.

pixl97 3y ago
Eh, this is much too simplistic of view. Yea jailbreaking your instance so it says "fuck the police" is one thing. Jailbreaking an LLM so if it looks at some particular picture and if that user happens to have XXX variable set it transfers $1,000,000 from their account to your account is a different story. These can both fall under the same wide umbrella.
- smoldesu 3y ago
  I think the preeminent issue is that AI is just not trustable for anything that matters. You cannot rely on an unaccountable stochastic model to handle important stuff like money.
  - thfuran 3y ago
    Fine, then it's sending 1,000,000 emails instead.
    - smoldesu 3y ago
      Well... I kinda think it's the same problem? You have designed a system in error if you are trusting an AI system to handle that sorta thing. We have programmable logic to handle payments and emails and all the stuff that's actually important. No self-respecting bank or email provider is going to implement an AI agent that has carte-blanche access to it's feature-set.
      - thfuran 3y ago
        Most emails don't originate from email providers.
        smoldesu 3y ago
        ...right. But AI did not invent the concept of automated email spam or wire fraud. I don't understand how this all relates to model jailbreaking.
        freedomben 3y ago
        If you're hitting some email providers API, then you don't need AI to do the spamming. I don't see that problem as any different from another anti-abuse responsibility of the email provider. AI could substantively rewrite the email for each mail so that it is hard to recognize as the same thing, but when the customer's recipients start marking things as spam it's gonna go through the existing processes.

pixl97 3y ago

The evolution of AI, especially as it becomes more powerful is going to be interesting. Eventually the biggest threat to AI will be other AI if we assume that AI capabilities are going to be come at least as good as human intellect. Once you get AI designed to self learn to attack, and ones that self learn how to defend (and the mix of both) and remove humans from this learning loop it's easy to see how things could go off the rails.

gizmo686 3y ago

We've known the current crop of AI is hilariously vulnerable to jailbreaking. Complete amateur humans are regularly able to do so with less than a days work. If the story was "AI driven fuzzer finds 0 day in popular application" no one would think twice. The only real story I see here is a reminder that LLMs are not fit for use on untrusted inputs; but, again, that is not new information.

sonicanatidae 3y ago
>The only real story I see here is a reminder that LLMs are not fit for use on untrusted inputs; but, again, that is not new information. And corps are ignoring that to roll out AI as fast as possible, to get in on the Gold Rush, because humans.
hobs 3y ago
That's pretty much what I was trying to say, I don't want to gatekeep anyone but I just see it as someone saying "this lock is super hard to pick so we should stop people from using pickguns!" when the core vulnerability can be exploited with just a little more time by any dingus who can watch a youtube video.

RandomLensman 3y ago

Any enormous model (especially created by some sort of optimization) will have a huge surface of problems that needs a second, independent layer to box it in. If the box cannot be described properly that will fail, of course.

svaha1728 3y ago
The problem is the training set. If the training set contains something disruptive or of value, there is a nonzero percent chance it will be the next token. Boxes can attenuate that risk, but cannot set it to zero.
- dragonwriter 3y ago
  > If the training set contains something disruptive or of value, there is a nonzero percent chance it will be the next token. Boxes can attenuate that risk, but cannot set it to zero. I mean, if it was as simple as you describe ("nonzero percent chance that it will be next token"), then boxes could easily prevent it, the framework hosting the LLM can adjust logit bias on each token, so if it there was a readily-determinable forbidden-next-token, could prevent it with no problem. Unfortunately, the reality is...more complex.