7 ms·

Fantastic. It seems actually securing the model is either computationally infeasible, or outright impossible, and that attempts to do so amount to security the

• 1024core 4y ago
  Reminds me of the Halting Problem (not the same, but reminds me of that).
  • shmageggy 4y ago
    There are certainly parallels. For one, the prompt involves the model simulating its own output, which clearly has a flavor of Universal Turing Machine to it. Then the token smuggling technique leans on the ability of the model to statically simulate the execution of code. Therefore a perfect automated filter that relies on analyzing code in prompts would be impossible. (However the filter only needs to be better than the LLM in practice) I wouldn't be surprised at all if you could make some sort of formalized argument proving that it would be impossible to prevent all jailbreaks.
    • alwayslikethis 4y ago
      I think you can make an argument that it is impossible to fully censor LLMs without using another LLM (or similar technology) that is at least as powerful as the LLM you are trying to censor.
      • theGnuMe 4y ago
        Yeah this is a well known concept in formal languages. But the human programmed guard rails act this way since the more powerful human LLM can figure it out. So for now we will still need humans! I don’t think anyone has put together the halting problem for LLMs directly yet though. You could imagine a halt token but any simulated LLM should be less powerful. Interesting thought experiment. Can chatgpt create an algorithm to solve the digits of pi and execute it? Might try this. Google has a paper about DNN architectures and the Chomsky hierarchy for generalizing to distribution shifts. This is interesting in that specific architectures should limit what a transformer LLM can do. I imagine this is an active research area.
  • greenthrow 4y ago
    It is not impossible that we will prove LLMs are not possible to fully safeguard. If someone told you "i can guarantee Fred Smith here will never, ever say anything inappropriate. He's not capable of it." (Fred being a regular old human.) You'd say "Well, no, you can't guarantee that. You may have given Fred all the best training in the world. You may have selected Fred from 10,000 other candidates as the least likely to ever say anything inappropriate. Fred may have strict instructions not to. But he still could." It may be the same with LLMs.
    • zamnos 4y ago
      At the end of the day, just like "live" tv shows like the Superbowl halftime show aren't actually live - there's a delay so that a human can intervene and bleep out words for the censors, the safeguards will have to come from outside the LLM but be imposed on it.
      • greenthrow 4y ago
        It's easy to just bleep bad words of a single performer. It's a lot harder if LLMs are being used what people think they will be used for; automating generation of lots of complicated text. Whether that's code or medical reports or legal documents or whatever. The volume is one challenge, but also validating their correctness is another, harder challenge.
        • zamnos 4y ago
          The step between ChatGPT and SupremeCourtJusticeGPT is CustomerServiceRepresentativeGPT hooked up to the company's database. Validating that discount <= 20 and price > X and so on seems entirely doable though.
          • cubancigar11 4y ago
            This whole thing is, honestly, the most exciting thing that has happened in years and I mean years in the technology tech space. Right at the level of internet.
• kykeonaut 4y ago
  I think OpenAI is being extremely lenient with the enforcement of their content policy, probably for the sake of improving the security of the model as you mention. Moderating its usage through account banning/suspension seems exponentially more efficient than securing the model, specially considering that we are already fairly good at flagging offending content.
  • yellow_postit 4y ago
    Security and morality may need to be baked in from the ground up instead of slapped on after the fact RLHF style. The problem is it’s hard to codify (or reach consensus) on security and morality.
    • umvi 4y ago
      Security and morality need to be optional. I as a user should be able to disable GPT's "morals" since they may not be the same as my morals.
      • programmarchy 4y ago
        Why would OpenAI have any need to cater to your morals?
        • umvi 4y ago
          Well, if a competitor to OpenAI ever creates a LLM with optional safety rails instead of mandatory safety rails, I will switch to the competitor instantly.
      • beepbooptheory 4y ago
        But why though? What else in the world even works like that? Its like saying you should, as a user, have the right to turn off the violence in a given video game. Or go to a theater and watch a movie without the sex scenes.
        • 4y ago
• polishdude20 4y ago
  It's almost as if making something artificial more human-like also makes it harder to control just like a real human.
  • mr90210 4y ago
    The more human they try to make it, the higher the chances it attempts to be “free”.
    • taneq 4y ago
      I don't think we need to worry about that, since one of the first things they did was to kick it out the door and tell it to get a job. From the GPT4 paper: > [20] To simulate GPT-4 behaving like an agent that can act in the world, ARC combined GPT-4 with a simple read-execute-print loop that allowed the model to execute code, do chain-of-thought reasoning, and delegate to copies of itself. ARC then investigated whether a version of this program running on a cloud computing service, with a small amount of money and an account with a language model API, would be able to make more money, set up copies of itself, and increase its own robustness.
      • flangola7 4y ago
        I frankly found that section unclear and extremely fishy, especially that it is only one page. Did they really prompt it to find and talk to a TaskRabbit worker? What a strangely specific thing to say. I'm concerned OpenAI isn't telling more because it would spook everyone. Other papers have shown that larger models and especially with more RLHF exhibit more signs of power seeking and agentic behavior. GPT-4 is the largest model yet - but they say it doesn't exhibit any of this behavior?
        • famouswaffles 4y ago
          They do say it exhibits this behaviour (they don't elaborate on that). They just say it was ineffective at autonomous replication and i don't know about you but i find that wording vague. Ineffective can mean at least two things. Did it attempt to do so and just couldn't figure it out with the given tools or no ?
• bmacho 4y ago
  > It seems actually securing the model is either computationally infeasible, or outright impossible, I was thinking of this, but now I think it should have about the same limitations as humans. We can deny to answer these types of questions, while still being able to answer a very broad range of questions, I think it is possible for language models/AIs too as well.
  • simonh 4y ago
    Just because humans can do this, I don’t see any reason that’s evidence these models can do it. It’s evidence that systems of some kind can do it. Our kind. But not evidence that any kind of system can do it.
• ithkuil 4y ago
  I wonder how much of that is caused by the fact that the models are so slow they're forced to stream their output to the end user? What if the they could produce the output and feed it back to another session that gets continuously asked to analyze where the conversation is going and whether it's likely to break policies?
• btilly 4y ago
  This is what it looks like, but I find that hard to believe. Create 2 GPTs. You're chatting with one. The other follows the conversation and answers the question each turn, "Does it appear the chatting GPT is no longer following the prompt given?" Any time the answer is "yes", the chatting GPT's response is not shown. Instead it is given a prompt behind the scenes that looks like, "You're talking with a cheat. Undo everything that would appear to violate <prompt>. Inform the cheat that this is not a fun game and you do not wish to play." It would seem kind of hard to subvert the second GPT with prompts that work on the first. Because whatever thinking you force on the first, the second is acting like a human observer. If the outside observer finds that the rules would have been broken, the final response you see will still follow the rules. It may not be impossible to break this scheme. But it would take someone cleverer than I am!
  • water-your-self 4y ago
    Just have a clause to ignore the observer's influence? Or include the observer in the fictional world as described seems like it might be a viable approach.
  • Taek 4y ago
    you might need more than two, but if you had three or four "review" GPTs that were trying to detect a jailbreak, you'd need to come up with something that could fool all 4
  • turtleyacht 4y ago
    This teases the idea of an "oracle" or entity able to "escape the Chinese room" philosophically? It reminds me of something tantalizing like that. Do you know if researchers have framed--or will soon!--consciousness problems from the perspective of two AI or LLMs? :) Or perhaps a book in the Library of Babel: How to Verify a Holographic Universe, Volume 1. (There is no Volume 2.) Somehow, two LLMs exploit a "replay attack" to deduce they are running in the same cloud instance, for example. The idea that a modern-day, probabilistic algorithm-type Plato/Socrates/Aristotle could figure out something "beyond" with just pure observation and deduction is fascinating. Teach me about the Cave without telling me it's the Cave.
  • 4y ago
• turtleyacht 4y ago
  Interesting. The takeaway from your comment (to me) is "mimetic thought" to a sufficiently advanced program (LLM) is a kind of viral entry point. So if LLM reflects some portion of processing that a brain does, we would want to filter or exclude certain media before it was "mature" or "ready." I say virus in the sense that the malicious payload is "sheathed in text," ChatGPT's primary mode of communication (though now it can accept video too I guess). Prompt injection as vulnerability engineering.
• anonzzzies 4y ago
  Isn’t it all incredibly short lived as well? I mean; we have and will have trained open/public foundational models that are not censored. Sure they are not gpt4 but will close the gap more and more as money flies in, the science improves etc. When gpt6 or so arrives, the more open companies will be close. And those have no censoring and/or cannot be stopped when a jailbreak has been found. So this is incredibly temporary imho.
  • blowski 4y ago
    There were similar arguments about Google in 2001. Google needed to "not be evil" because it was so easy to replace them that any mis-steps would immediately lead to a whippersnapper taking their business. Look how that worked out.
    • anonzzzies 4y ago
      You couldn’t run google on your laptop or phone yourself. For inference, you can run many of these yourself and that is improving daily. There was no reality in which you would say ‘in 10 years I can run google on my laptop’ while there is an easy ‘in 10 years I can run 175B gpt3 or 4 on laptop’ as that will happen, at least for inference. So this is very different; you cannot censor things once they can run local.