4 ms·

Reminds me of the Halting Problem (not the same, but reminds me of that).

shmageggy 4y ago
There are certainly parallels. For one, the prompt involves the model simulating its own output, which clearly has a flavor of Universal Turing Machine to it. Then the token smuggling technique leans on the ability of the model to statically simulate the execution of code. Therefore a perfect automated filter that relies on analyzing code in prompts would be impossible. (However the filter only needs to be better than the LLM in practice) I wouldn't be surprised at all if you could make some sort of formalized argument proving that it would be impossible to prevent all jailbreaks.
- alwayslikethis 4y ago
  I think you can make an argument that it is impossible to fully censor LLMs without using another LLM (or similar technology) that is at least as powerful as the LLM you are trying to censor.
  - theGnuMe 4y ago
    Yeah this is a well known concept in formal languages. But the human programmed guard rails act this way since the more powerful human LLM can figure it out. So for now we will still need humans! I don’t think anyone has put together the halting problem for LLMs directly yet though. You could imagine a halt token but any simulated LLM should be less powerful. Interesting thought experiment. Can chatgpt create an algorithm to solve the digits of pi and execute it? Might try this. Google has a paper about DNN architectures and the Chomsky hierarchy for generalizing to distribution shifts. This is interesting in that specific architectures should limit what a transformer LLM can do. I imagine this is an active research area.
greenthrow 4y ago
It is not impossible that we will prove LLMs are not possible to fully safeguard. If someone told you "i can guarantee Fred Smith here will never, ever say anything inappropriate. He's not capable of it." (Fred being a regular old human.) You'd say "Well, no, you can't guarantee that. You may have given Fred all the best training in the world. You may have selected Fred from 10,000 other candidates as the least likely to ever say anything inappropriate. Fred may have strict instructions not to. But he still could." It may be the same with LLMs.
- zamnos 4y ago
  At the end of the day, just like "live" tv shows like the Superbowl halftime show aren't actually live - there's a delay so that a human can intervene and bleep out words for the censors, the safeguards will have to come from outside the LLM but be imposed on it.
  - greenthrow 4y ago
    It's easy to just bleep bad words of a single performer. It's a lot harder if LLMs are being used what people think they will be used for; automating generation of lots of complicated text. Whether that's code or medical reports or legal documents or whatever. The volume is one challenge, but also validating their correctness is another, harder challenge.
    - zamnos 4y ago
      The step between ChatGPT and SupremeCourtJusticeGPT is CustomerServiceRepresentativeGPT hooked up to the company's database. Validating that discount <= 20 and price > X and so on seems entirely doable though.
      - cubancigar11 4y ago
        This whole thing is, honestly, the most exciting thing that has happened in years and I mean years in the technology tech space. Right at the level of internet.
- taneq 4y ago
  I'd wager it is with any sufficiently intelligent system. Once it has agency (or can sufficiently well simulate something with agency, which is the same thing) you can't ever be 100% certain what it will do beforehand.

There are certainly parallels. For one, the prompt involves the model simulating its own output, which clearly has a flavor of Universal Turing Machine to it. Then the token smuggling technique leans on the ability of the model to statically simulate the execution of code. Therefore a perfect automated filter that relies on analyzing code in prompts would be impossible. (However the filter only needs to be better than the LLM in practice) I wouldn't be surprised at all if you could make some sort of formalized argument proving that it would be impossible to prevent all jailbreaks.

alwayslikethis 4y ago
I think you can make an argument that it is impossible to fully censor LLMs without using another LLM (or similar technology) that is at least as powerful as the LLM you are trying to censor.
- theGnuMe 4y ago
  Yeah this is a well known concept in formal languages. But the human programmed guard rails act this way since the more powerful human LLM can figure it out. So for now we will still need humans! I don’t think anyone has put together the halting problem for LLMs directly yet though. You could imagine a halt token but any simulated LLM should be less powerful. Interesting thought experiment. Can chatgpt create an algorithm to solve the digits of pi and execute it? Might try this. Google has a paper about DNN architectures and the Chomsky hierarchy for generalizing to distribution shifts. This is interesting in that specific architectures should limit what a transformer LLM can do. I imagine this is an active research area.

greenthrow 4y ago

It is not impossible that we will prove LLMs are not possible to fully safeguard. If someone told you "i can guarantee Fred Smith here will never, ever say anything inappropriate. He's not capable of it." (Fred being a regular old human.) You'd say "Well, no, you can't guarantee that. You may have given Fred all the best training in the world. You may have selected Fred from 10,000 other candidates as the least likely to ever say anything inappropriate. Fred may have strict instructions not to. But he still could." It may be the same with LLMs.

zamnos 4y ago
At the end of the day, just like "live" tv shows like the Superbowl halftime show aren't actually live - there's a delay so that a human can intervene and bleep out words for the censors, the safeguards will have to come from outside the LLM but be imposed on it.
- greenthrow 4y ago
  It's easy to just bleep bad words of a single performer. It's a lot harder if LLMs are being used what people think they will be used for; automating generation of lots of complicated text. Whether that's code or medical reports or legal documents or whatever. The volume is one challenge, but also validating their correctness is another, harder challenge.
  - zamnos 4y ago
    The step between ChatGPT and SupremeCourtJusticeGPT is CustomerServiceRepresentativeGPT hooked up to the company's database. Validating that discount <= 20 and price > X and so on seems entirely doable though.
    - cubancigar11 4y ago
      This whole thing is, honestly, the most exciting thing that has happened in years and I mean years in the technology tech space. Right at the level of internet.
taneq 4y ago
I'd wager it is with any sufficiently intelligent system. Once it has agency (or can sufficiently well simulate something with agency, which is the same thing) you can't ever be 100% certain what it will do beforehand.