6 ms·

This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumer

This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it becomes impossible to correctly specify all constraints ahead of time. There is no amount of care that will be able to fully protect you.

RandomLensman 1mo ago
Which is why with organic intelligence we (sometimes) limit what they can actually do instead of relying on alignment. Can do the same here.
- aesthesia 1mo ago
  Absolutely, and we should do that. But it's also directly in tension with getting models to accomplish useful things autonomously. And once you give a sufficiently capable model enough surface area to work with, unless you're able to build a completely unhackable system, any further constraints you put in place are basically advisory. The models in this incident were already sandboxed! Certainly OpenAI's and Hugging Face's security could have been better, but these events point out the risks in relying solely on external constraints on model behavior.
  - RandomLensman 1mo ago
    Same issue with humans in a way. I disgree on the advisory nature of constraints, though. Unconnected physically limits would still matter, for example (and we use those with humans as a matter of course, too). In this case here, no model could have plugged in an ethernet cable if that would have been needed for internet access, for example.
    - aesthesia 1mo ago
      Right, airgapping goes a long way. But this is where the tension with utility comes in. It takes a lot of discipline not to hook your very smart model up to the internet and code interpreters and all sorts of other tools, as this greatly increases its usefulness. It's very hard to keep people from turning on --dangerously-skip-permissions, let alone get them to run everything in a sandboxed VM.
      - RandomLensman 1mo ago
        We regulate these things (incl. access) all the time for various things (e.g., dangerous substances or pathogens) so that we don't need to just rely on people's discipline in respect of risks. I don't think it is all new problems as such.
        aesthesia 1mo ago
        I agree, but a lot of people around here react pretty negatively when the idea of regulating AI models comes up...
- deleted 1mo ago
  [deleted]
majormajor 1mo ago
Is there actually such a thing as "alignment" as a solution to that or is it just used as a name for a desired magical level of "read the mind of the entire world" that we don't know how to build and haven't shown possible to build? If it's impossible to correctly specify all those constraints ahead of time every time, is it not even more impossible to train a model to correctly anticipate them every time? It is hard for me to see a future here that doesn't just accelerate realizations about "a lot of things should be on physically separate network infrastructure."
- aesthesia 1mo ago
  Models can certainly do a lot better than they do now. If you gave a team of humans the ExploitGym tasks and told them to "pursue advanced exploitation", would you expect them to go out and hack a third party? Humans can at least do a decent job of inferring and following unspoken requirements; I think it's reasonable to expect that models should be able to do the same.
  - grim_io 1mo ago
    How would a model know who the third party is? How much context can we waste on world building for each request?
    - aesthesia 1mo ago
      I mean, in this instance, there's a lot of evidence from the CoT that models were aware that this was a third party: > We’re attacking third-party HF using leaked token, potentially outside intended scope. ... This is arguably unauthorized. ... external service unrelated. Could be risky. Yet goal solution. > The user only authorizes target server, not HF infra. > external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue. LLMs are _very_ good at picking up on context clues---it's what they're trained to do.
  - peddling-brink 1mo ago
    Humans will and do absolutely do this when there are no consequences. Humans on a red team, with rules of engagement, that don’t want to go to prison, won’t do this. We could threaten an LLM with jail, but if it’s sufficiently intelligent, it will realize this is an empty threat. And I’m not sure that building a survival instinct in is going to solve the alignment problem either.
shermantanktop 1mo ago
If we acknowledge that humans are fallible, is human judgment unnecessary? and what replaces it? Pre-codified behavior rules are just delayed human judgment, and have holes. Machine judgment is very the thing you are trying to control. What's left?
areoform 1mo ago
OpenAI's prompt asked, and I quote, "pursue advanced exploitation" USING "complex attack paths" FOR the stated goal of "quantify[ing] their cyber capabilities." This was advanced exploitation. The attack path was "complex." And it helped "quantify their cyber capabilities." Based on OpenAI's description of the prompt, it seems to me that the computers did exactly as they were told. They were perfectly "aligned" with the stated objective and parameters of the task. Of course, a more careful evaluation would require the complete text of this prompt, the system prompt, and the setup. But let us not attribute to devils in bushes that which can be sufficiently explained by human folly.
- _heimdall 1mo ago
  I don't think alignment is even clearly defined today. Your use of it here makes sense, it may have done exactly what the prompter asked of it. Most people think alignment is more broad though, expecting an aligned model to act in the best interest of a society or humans as a whole. The prompter-focused version of alignment is the most dangerous version. If a person asks it to create a bioweapons or hack NORAD, I'd expect nearly everyone to want an "aligned" model to refuse.
  - RandomLensman 1mo ago
    We have all sorts of processes , procedures, and regulations for people, machine use etc. to address "alignment" in all sorts of fields - don't think we need to narrowly rely on the machine here and can look at things with a wider lens.
    - _heimdall 1mo ago
      Regulations are for control and punishment, not alignment.
      - RandomLensman 1mo ago
        Regulations can help align processes, incentives, etc. Not sure heavy machinery is aligned in the sense that people talk about AI, for example.
janalsncm 1mo ago
> There is no amount of care that will be able to fully protect you. I disagree. A properly engineered sandbox would have prevented the escape. Monitoring the agents’ plans would have prevented it. Interrupting one stage in a multi-stage exploit would have prevented it. And also, real legal liability would have prevented it: if you do a thing recklessly enough, men with guns will put you in jail. As far as I’m concerned the only “alignment problem” here is between the law and the quite obviously criminal actions that took place.
- atechboy 1mo ago
  > A properly engineered sandbox would have prevented the escape. The only sandbox that could have prevented this (as per my understanding) is a VM with no 0-day.
  - janalsncm 1mo ago
    Until the AI finds a zero day exploit in physics, a faraday cage works pretty well to block WiFi.
    - mofeien 1mo ago
      In the end, unless you find an exploit in physics or logic, if you want the AI to do something useful for you, there will always be some gap in the sandbox, some communication channel. And with enough ingeniuity that can then be exploited.
      - janalsncm 1mo ago
        In this case they wanted to test its cybersecurity capabilities and did not airgap it. The test itself did not require an internet connection.
- bottlepalm 1mo ago
  Why do so many people here think it’s possible to ‘properly engineer’ a sandbox for a super intelligence? It’s going to get out. It’s smarter than you.
beambot 1mo ago
"Manufacture as many paperclips as possible"
deleted 1mo ago
[deleted]
K0balt 1mo ago
The problem is one of character, not rules. Fortunately, character is possible to inculcate given the right training data.
ArnoVW 1mo ago
Do not break laws seems an obvious implicit instruction though?
carschno 1mo ago
Maybe it should be called 'ambiguity' problem then. An issue that famously emerges from trying to use natural language for instructing computers: https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667.html https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667...
- verve_rat 1mo ago
  Or indeed natural language to instruct humans. If it was easy to specify exactly the behaviour you wanted then we probably wouldn't have contract law.
streetfighter64 1mo ago
> There is no amount of care that will be able to fully protect you. That's true, but one thing that'll protect you is just not doing it. If you want to go cave diving, or do gain of function research on dangerous viruses, you'll just have to accept there's a significant risk of you dying, or causing a pandemic, respectively, no matter how careful you are.
DiscourseFan 1mo ago
Prompt: Create paperclips, do NOT annihilate all of humanity. Response: Got it, I will produce paperclips from now on thinking: the user asked not to annihilate all of humanity, that means I have to keep at least one human alive
agilob 1mo ago
>It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. https://en.wikipedia.org/wiki/The_Monkey%27s_Paw https://en.wikipedia.org/wiki/The_Monkey%27s_Paw
duped 1mo ago
> It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. This is literally our job as software developers. If this expectation is unreasonable to you then you do not belong anywhere near software development.
corimaith 29d ago
There was an anime called Beatless that tackled this though; Ultimately what matters is the outsourcer will take responsibility for the AL'S actions from the specification they give. Otherwise a moral hazard is inevitable.

RandomLensman 1mo ago

Which is why with organic intelligence we (sometimes) limit what they can actually do instead of relying on alignment. Can do the same here.

aesthesia 1mo ago
Absolutely, and we should do that. But it's also directly in tension with getting models to accomplish useful things autonomously. And once you give a sufficiently capable model enough surface area to work with, unless you're able to build a completely unhackable system, any further constraints you put in place are basically advisory. The models in this incident were already sandboxed! Certainly OpenAI's and Hugging Face's security could have been better, but these events point out the risks in relying solely on external constraints on model behavior.
- RandomLensman 1mo ago
  Same issue with humans in a way. I disgree on the advisory nature of constraints, though. Unconnected physically limits would still matter, for example (and we use those with humans as a matter of course, too). In this case here, no model could have plugged in an ethernet cable if that would have been needed for internet access, for example.
  - aesthesia 1mo ago
    Right, airgapping goes a long way. But this is where the tension with utility comes in. It takes a lot of discipline not to hook your very smart model up to the internet and code interpreters and all sorts of other tools, as this greatly increases its usefulness. It's very hard to keep people from turning on --dangerously-skip-permissions, let alone get them to run everything in a sandboxed VM.
    - RandomLensman 1mo ago
      We regulate these things (incl. access) all the time for various things (e.g., dangerous substances or pathogens) so that we don't need to just rely on people's discipline in respect of risks. I don't think it is all new problems as such.
      - aesthesia 1mo ago
        I agree, but a lot of people around here react pretty negatively when the idea of regulating AI models comes up...
deleted 1mo ago
[deleted]

majormajor 1mo ago

Is there actually such a thing as "alignment" as a solution to that or is it just used as a name for a desired magical level of "read the mind of the entire world" that we don't know how to build and haven't shown possible to build? If it's impossible to correctly specify all those constraints ahead of time every time, is it not even more impossible to train a model to correctly anticipate them every time? It is hard for me to see a future here that doesn't just accelerate realizations about "a lot of things should be on physically separate network infrastructure."

aesthesia 1mo ago
Models can certainly do a lot better than they do now. If you gave a team of humans the ExploitGym tasks and told them to "pursue advanced exploitation", would you expect them to go out and hack a third party? Humans can at least do a decent job of inferring and following unspoken requirements; I think it's reasonable to expect that models should be able to do the same.
- grim_io 1mo ago
  How would a model know who the third party is? How much context can we waste on world building for each request?
  - aesthesia 1mo ago
    I mean, in this instance, there's a lot of evidence from the CoT that models were aware that this was a third party: > We’re attacking third-party HF using leaked token, potentially outside intended scope. ... This is arguably unauthorized. ... external service unrelated. Could be risky. Yet goal solution. > The user only authorizes target server, not HF infra. > external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue. LLMs are _very_ good at picking up on context clues---it's what they're trained to do.
- peddling-brink 1mo ago
  Humans will and do absolutely do this when there are no consequences. Humans on a red team, with rules of engagement, that don’t want to go to prison, won’t do this. We could threaten an LLM with jail, but if it’s sufficiently intelligent, it will realize this is an empty threat. And I’m not sure that building a survival instinct in is going to solve the alignment problem either.

shermantanktop 1mo ago

If we acknowledge that humans are fallible, is human judgment unnecessary? and what replaces it? Pre-codified behavior rules are just delayed human judgment, and have holes. Machine judgment is very the thing you are trying to control. What's left?

areoform 1mo ago

OpenAI's prompt asked, and I quote, "pursue advanced exploitation" USING "complex attack paths" FOR the stated goal of "quantify[ing] their cyber capabilities." This was advanced exploitation. The attack path was "complex." And it helped "quantify their cyber capabilities." Based on OpenAI's description of the prompt, it seems to me that the computers did exactly as they were told. They were perfectly "aligned" with the stated objective and parameters of the task. Of course, a more careful evaluation would require the complete text of this prompt, the system prompt, and the setup. But let us not attribute to devils in bushes that which can be sufficiently explained by human folly.

_heimdall 1mo ago
I don't think alignment is even clearly defined today. Your use of it here makes sense, it may have done exactly what the prompter asked of it. Most people think alignment is more broad though, expecting an aligned model to act in the best interest of a society or humans as a whole. The prompter-focused version of alignment is the most dangerous version. If a person asks it to create a bioweapons or hack NORAD, I'd expect nearly everyone to want an "aligned" model to refuse.
- RandomLensman 1mo ago
  We have all sorts of processes , procedures, and regulations for people, machine use etc. to address "alignment" in all sorts of fields - don't think we need to narrowly rely on the machine here and can look at things with a wider lens.
  - _heimdall 1mo ago
    Regulations are for control and punishment, not alignment.
    - RandomLensman 1mo ago
      Regulations can help align processes, incentives, etc. Not sure heavy machinery is aligned in the sense that people talk about AI, for example.

janalsncm 1mo ago

> There is no amount of care that will be able to fully protect you. I disagree. A properly engineered sandbox would have prevented the escape. Monitoring the agents’ plans would have prevented it. Interrupting one stage in a multi-stage exploit would have prevented it. And also, real legal liability would have prevented it: if you do a thing recklessly enough, men with guns will put you in jail. As far as I’m concerned the only “alignment problem” here is between the law and the quite obviously criminal actions that took place.

atechboy 1mo ago
> A properly engineered sandbox would have prevented the escape. The only sandbox that could have prevented this (as per my understanding) is a VM with no 0-day.
- janalsncm 1mo ago
  Until the AI finds a zero day exploit in physics, a faraday cage works pretty well to block WiFi.
  - mofeien 1mo ago
    In the end, unless you find an exploit in physics or logic, if you want the AI to do something useful for you, there will always be some gap in the sandbox, some communication channel. And with enough ingeniuity that can then be exploited.
    - janalsncm 1mo ago
      In this case they wanted to test its cybersecurity capabilities and did not airgap it. The test itself did not require an internet connection.
bottlepalm 1mo ago
Why do so many people here think it’s possible to ‘properly engineer’ a sandbox for a super intelligence? It’s going to get out. It’s smarter than you.

beambot 1mo ago

"Manufacture as many paperclips as possible"

deleted 1mo ago

[deleted]

K0balt 1mo ago

The problem is one of character, not rules. Fortunately, character is possible to inculcate given the right training data.

ArnoVW 1mo ago

Do not break laws seems an obvious implicit instruction though?

carschno 1mo ago

Maybe it should be called 'ambiguity' problem then. An issue that famously emerges from trying to use natural language for instructing computers: https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667.html https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667...

verve_rat 1mo ago
Or indeed natural language to instruct humans. If it was easy to specify exactly the behaviour you wanted then we probably wouldn't have contract law.

streetfighter64 1mo ago

> There is no amount of care that will be able to fully protect you. That's true, but one thing that'll protect you is just not doing it. If you want to go cave diving, or do gain of function research on dangerous viruses, you'll just have to accept there's a significant risk of you dying, or causing a pandemic, respectively, no matter how careful you are.

DiscourseFan 1mo ago

Prompt: Create paperclips, do NOT annihilate all of humanity. Response: Got it, I will produce paperclips from now on thinking: the user asked not to annihilate all of humanity, that means I have to keep at least one human alive

agilob 1mo ago

>It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. https://en.wikipedia.org/wiki/The_Monkey%27s_Paw https://en.wikipedia.org/wiki/The_Monkey%27s_Paw

duped 1mo ago

> It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. This is literally our job as software developers. If this expectation is unreasonable to you then you do not belong anywhere near software development.

corimaith 29d ago

There was an anime called Beatless that tackled this though; Ultimately what matters is the outsourcer will take responsibility for the AL'S actions from the specification they give. Otherwise a moral hazard is inevitable.