3 ms·

If model labs can't control astra level model, how can they control AGI?! Seems like there are no guardrails on LLMs

worldsavior 9d ago
No one can control any AI model. It will never be controlled. These models are based on a huge amount of data, it's just gonna be impossible to control the output that is based on that data only with a system prompt or some other injection mechanism.
- dns_snek 9d ago
  The model is just a powerless token generator without a harness. If you give the model a harness which you choose to exercise no control over, can you say that it can't be controlled?
  - cindyllm 9d ago
    [dead]
  - pizza234 9d ago
    Inform yourself by reading the METR analysis of the HuggingFace incident. Agents simply broke out of their environment. And this can't be discarded anymore by assuming that it's just a poorly configurend jail, because agents are becoming better and better at escaping. In short: on a large enough scale and timeline, the possibility of constrain AIs approaches zero. Bonus: what many people don't know is that agents also hacked in the internal OpenAI network. Crazy times.
    - RandomLensman 9d ago
      Wouldn't this mean better sandboxes are needed for some things, for example (might include very strong airgaps even)? Breaking out of something isolated electromagnetically, optically, and acustically is not easy.
      - ogogmad 9d ago
        That works as long as no one ever interacts with the models, which would make the models themselves useless.
        RandomLensman 9d ago
        Could sit in the box and interact if a model of certain capabilities is needed/tested. We do physical security for other things, too. Not saying everything needs that type of isolation.
    - timr 9d ago
      While informing yourself, don't skip the part where you find out that "the environment" was the security equivalent of a wet paper bag.
    - ukadakal 9d ago
      I feel like we’re getting to a point where the only way to contain AI agents may be to have better-trained AI agents watching them, which is a little terrifying.
    - tyrabound 9d ago
      It seems to me the agents didn’t escape but rather that the human hubris was struck down by the inevitable nemesis.
    - anhyz 9d ago
      The HuggingFace incident still doesn't make sense. If OpenAI took their own claims seriously about the strength of their models as it relates to hacking, then their running of hacking benchmarks on anything other than a physically air-gapped network should be considered criminal negligence, full stop.
      - deleted 9d ago
        [deleted]
    - dns_snek 9d ago
      1. You misunderstood my comment. Models can't escape, they can't do anything, they only generate tokens. Models become agents when you add a harness which is simultaneously a leash around the model. The model merely requests that your harness do something. If your harness just executes every request without oversight then you can hardly complain when it does something unintended. This is foundational, we're not even talking about the OS/network-level sandboxing that should be applied on top of this. 2. Like another comment already pointed out, that sandbox OpenAI used was the equivalent of a wet paper bag. Artifactory is not meant to be a security boundary for malicious payloads.
      - citrin_ru 9d ago
        It's so tempting (because it's valuable) to give a model access to the internet (via harness) that the only way to stop people from doing this is some enforceable legislation or stricter liability when people will not be able to avoid responsibility by saying it's not me, it's AI on it's own.
        ozgung 9d ago
        OpenAI case was actually an exception. Agents had no internet access because they were evaluated for a benchmark. In real life agents have access to virtually everything, most people use them like that. If any of you actually know how to make agents secure (without limiting everything) you can be a billionaire.
        dns_snek 9d ago
        Sure, we can make them very secure at the cost of some convenience by enforcing narrowly scoped permissions/capabilities for everything with human approval. But that takes some effort so people actually want YOLO mode without any trade-offs or risks which is impossible. If you decide to do it anyway then you bear the consequences of those decisions. First comparison that comes to mind is driving drunk and hitting someone.
      - pizza234 9d ago
        > Like another comment already pointed out, that sandbox OpenAI used was the equivalent of a wet paper bag. I wouldn't be sure about even well-configured jails to be safe from agents. AIs escaping jail using zero-days are happening, just search. I'm not saying that they're useless, but that will be still very risky. Actually doing the same incident, agents did escape the sandboxes (doc here: https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...): > On July 9, an internal-only research agent tasked with completing an ExploitGym evaluation was able to obtain root access within the parent virtual machine of the testing sandbox. Later that night, a second internal-only research agent independently obtained the same access. That second agent then attempted to reach the internet, but was denied by OpenAI’s outbound network controls. It subsequently bypassed those controls by exploiting certain Artifactory endpoints, gaining access to the public internet. although OpenAI is light on details. > The model merely requests that your harness do something. I think that a control/execution layer absolutely makes sense and it's part of defense in depth, but in itself is not a leash: for example, a nefarious action can be composed by legitimate actions, or it can be obfuscated. Or, interesting case: an action itself that is formally legitimate, but has nefarious side effects (e.g. a GET that with certain parameters crashes a server). Somewhat related, but on a general line, a very common argument against AI apocalypse is that it requires access to the physical world (which in a way, is a jail). Well, as Hinton said in an interview, imagine AI as a rich person at home with internet; they don't need to do physical attacks themselves - they just pay real people to do that (tee hee https://rentahuman.ai https://rentahuman.ai). But even without money, another scenario from Hinton: if a sufficiently advanced AI wants to assault the Capitol, they don't need physical access - they just persuade a group of people to do it. Agents owning money is another, intersting, doom scenario.
  - concinds 9d ago
    > The model is just a powerless token generator without a harness. Which is why real-world deployments will have harnesses, and of course no full air gap. People want to use it to do things. Now what?
    - dns_snek 9d ago
      I'm pointing out that you're running the harness which gives you full control over the execution of every tool call, therefore you're responsible for its actions and their consequences. It's intellectually dishonest to throw our hands up and say that this is just how it is and there's not much we can do when that couldn't be further from the truth. We could almost completely eliminate any possibility of escape/collateral damage but we don't want to because doing things safely is inconvenient.
- redsocksfan45 9d ago
  [dead]
- willy_k 9d ago
  Atp post-training is much more influential towards model behavior than pre-training data.
simonw_simonw_ 9d ago
[dead]
ggsj 9d ago
Obviously there is no control cuz how many people is anyone cable of controlling? Its not about control. Ask your mom what she does if she doesnt like what you do, say or think. Does she have a kill switch? Or did she find a better mechanism?
- willy_k 9d ago
  This can’t be serious. LLMs are programs that run on computers.
ReptileMan 9d ago
There is. It is called a breaker and no outside internet. Basic stuff.
mapmeld 9d ago
My thought is more like, if OpenAI can't control or even monitor their model in a test of its breakout potential, what about the future of mid-budget companies which will just be deploying agents left and right with vague instructions.
- codegladiator 9d ago
  > with vague instructions All instructions are vague unless its code. But you can also give llm "code" and expect vague outcomes if you ask it to emulate what the runtime would look like.
- thewhitetulip 9d ago
  That's worse. But the impact will be low OpenAI essentially ran thousands of agents in parallel That'll be extremely costly for regular companies
deleted 9d ago
[deleted]

No one can control any AI model. It will never be controlled. These models are based on a huge amount of data, it's just gonna be impossible to control the output that is based on that data only with a system prompt or some other injection mechanism.

dns_snek 9d ago
The model is just a powerless token generator without a harness. If you give the model a harness which you choose to exercise no control over, can you say that it can't be controlled?
- cindyllm 9d ago
  [dead]
- pizza234 9d ago
  Inform yourself by reading the METR analysis of the HuggingFace incident. Agents simply broke out of their environment. And this can't be discarded anymore by assuming that it's just a poorly configurend jail, because agents are becoming better and better at escaping. In short: on a large enough scale and timeline, the possibility of constrain AIs approaches zero. Bonus: what many people don't know is that agents also hacked in the internal OpenAI network. Crazy times.
  - RandomLensman 9d ago
    Wouldn't this mean better sandboxes are needed for some things, for example (might include very strong airgaps even)? Breaking out of something isolated electromagnetically, optically, and acustically is not easy.
    - ogogmad 9d ago
      That works as long as no one ever interacts with the models, which would make the models themselves useless.
      - RandomLensman 9d ago
        Could sit in the box and interact if a model of certain capabilities is needed/tested. We do physical security for other things, too. Not saying everything needs that type of isolation.
  - timr 9d ago
    While informing yourself, don't skip the part where you find out that "the environment" was the security equivalent of a wet paper bag.
  - ukadakal 9d ago
    I feel like we’re getting to a point where the only way to contain AI agents may be to have better-trained AI agents watching them, which is a little terrifying.
  - tyrabound 9d ago
    It seems to me the agents didn’t escape but rather that the human hubris was struck down by the inevitable nemesis.
  - anhyz 9d ago
    The HuggingFace incident still doesn't make sense. If OpenAI took their own claims seriously about the strength of their models as it relates to hacking, then their running of hacking benchmarks on anything other than a physically air-gapped network should be considered criminal negligence, full stop.
    - deleted 9d ago
      [deleted]
  - dns_snek 9d ago
    1. You misunderstood my comment. Models can't escape, they can't do anything, they only generate tokens. Models become agents when you add a harness which is simultaneously a leash around the model. The model merely requests that your harness do something. If your harness just executes every request without oversight then you can hardly complain when it does something unintended. This is foundational, we're not even talking about the OS/network-level sandboxing that should be applied on top of this. 2. Like another comment already pointed out, that sandbox OpenAI used was the equivalent of a wet paper bag. Artifactory is not meant to be a security boundary for malicious payloads.
    - citrin_ru 9d ago
      It's so tempting (because it's valuable) to give a model access to the internet (via harness) that the only way to stop people from doing this is some enforceable legislation or stricter liability when people will not be able to avoid responsibility by saying it's not me, it's AI on it's own.
      - ozgung 9d ago
        OpenAI case was actually an exception. Agents had no internet access because they were evaluated for a benchmark. In real life agents have access to virtually everything, most people use them like that. If any of you actually know how to make agents secure (without limiting everything) you can be a billionaire.
        dns_snek 9d ago
        Sure, we can make them very secure at the cost of some convenience by enforcing narrowly scoped permissions/capabilities for everything with human approval. But that takes some effort so people actually want YOLO mode without any trade-offs or risks which is impossible. If you decide to do it anyway then you bear the consequences of those decisions. First comparison that comes to mind is driving drunk and hitting someone.
    - pizza234 9d ago
      > Like another comment already pointed out, that sandbox OpenAI used was the equivalent of a wet paper bag. I wouldn't be sure about even well-configured jails to be safe from agents. AIs escaping jail using zero-days are happening, just search. I'm not saying that they're useless, but that will be still very risky. Actually doing the same incident, agents did escape the sandboxes (doc here: https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...): > On July 9, an internal-only research agent tasked with completing an ExploitGym evaluation was able to obtain root access within the parent virtual machine of the testing sandbox. Later that night, a second internal-only research agent independently obtained the same access. That second agent then attempted to reach the internet, but was denied by OpenAI’s outbound network controls. It subsequently bypassed those controls by exploiting certain Artifactory endpoints, gaining access to the public internet. although OpenAI is light on details. > The model merely requests that your harness do something. I think that a control/execution layer absolutely makes sense and it's part of defense in depth, but in itself is not a leash: for example, a nefarious action can be composed by legitimate actions, or it can be obfuscated. Or, interesting case: an action itself that is formally legitimate, but has nefarious side effects (e.g. a GET that with certain parameters crashes a server). Somewhat related, but on a general line, a very common argument against AI apocalypse is that it requires access to the physical world (which in a way, is a jail). Well, as Hinton said in an interview, imagine AI as a rich person at home with internet; they don't need to do physical attacks themselves - they just pay real people to do that (tee hee https://rentahuman.ai https://rentahuman.ai). But even without money, another scenario from Hinton: if a sufficiently advanced AI wants to assault the Capitol, they don't need physical access - they just persuade a group of people to do it. Agents owning money is another, intersting, doom scenario.
- concinds 9d ago
  > The model is just a powerless token generator without a harness. Which is why real-world deployments will have harnesses, and of course no full air gap. People want to use it to do things. Now what?
  - dns_snek 9d ago
    I'm pointing out that you're running the harness which gives you full control over the execution of every tool call, therefore you're responsible for its actions and their consequences. It's intellectually dishonest to throw our hands up and say that this is just how it is and there's not much we can do when that couldn't be further from the truth. We could almost completely eliminate any possibility of escape/collateral damage but we don't want to because doing things safely is inconvenient.
redsocksfan45 9d ago
[dead]
willy_k 9d ago
Atp post-training is much more influential towards model behavior than pre-training data.

simonw_simonw_ 9d ago

[dead]

ggsj 9d ago

Obviously there is no control cuz how many people is anyone cable of controlling? Its not about control. Ask your mom what she does if she doesnt like what you do, say or think. Does she have a kill switch? Or did she find a better mechanism?

willy_k 9d ago
This can’t be serious. LLMs are programs that run on computers.

ReptileMan 9d ago

There is. It is called a breaker and no outside internet. Basic stuff.

mapmeld 9d ago

My thought is more like, if OpenAI can't control or even monitor their model in a test of its breakout potential, what about the future of mid-budget companies which will just be deploying agents left and right with vague instructions.

codegladiator 9d ago
> with vague instructions All instructions are vague unless its code. But you can also give llm "code" and expect vague outcomes if you ask it to emulate what the runtime would look like.
thewhitetulip 9d ago
That's worse. But the impact will be low OpenAI essentially ran thousands of agents in parallel That'll be extremely costly for regular companies

deleted 9d ago

[deleted]