4 ms·

It's more a matter of values this AI will hold than its level of intelligence. Boström argues that containment isn't really possible.

by blfr 12y ago

FatalLogic 12y ago
Is there any sure way to stop the super intelligent AI from changing its own inherent values though? I guess a problem with containment is that unless it's mathematically proven to be impervious, the super intelligent AI could have figured out that it's being contained and started to learn about the real world and how to manipulate it, while the humans outside still think they're safe.
- ThrustVectoring 12y ago
  Changing your inherent values is a poor way to go about achieving your current set of values. The biggest issue is ensuring that everything we care about gets encapsulated as a value of the AI. If not, we wind up with things like "set off a bomb in the basement to more efficiently get people out of a burning building."
- JamesArgo 12y ago
  It's assumed a rational agent judges its actions by how likely said action are to align the world with their fundamental values. For this reason, it is argued, a rational agent will not change its terminal goals. For example, suppose we build an agent that desires only to make shoes. That is, its ideal universe would consist of all available matter being converted into sneakers. Would it realize this is stupid? No. It judges it actions and thoughts only by how many shoes they are likely to bring into being. Any thought against this would be judged, correctly, as a negative contribution to shoe production and so wouldn’t be implemented, and likely would remain unthought. The best paper on this by far it Basic AI Drives by Stephen M. OMOHUNDRO: http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf http://selfawaresystems.files.wordpress.com/2008/01/ai_drive... I urge you to read it. It really is a wonderful paper.
  - TeMPOraL 12y ago
    The real problem is that human values are incredibly complex and messy, and any deviation from them multiplied by the power of a superintelligence would most likely doom us forever.
- eli_gottlieb 12y ago
  > Is there any sure way to stop the super intelligent AI from changing its own inherent values though? Yes: give it inherent values. >I guess a problem with containment is that unless it's mathematically proven to be impervious, AGI, when it eventually exists, must run according to the basic laws of learning. If you don't give it evidence that's more likely in possible-worlds with an external world than without one, it won't posit an external world. Stopping that information leak isn't that hard, provided you have full control over the agent-environment surface in the first place. It does, however, make your "agent" completely useless for anything beyond playing Atari games.
rl3 12y ago
> Boström argues that containment isn't really possible. He never argued that containment isn't possible, just that it's incredibly dangerous. In fact, in Superintelligence, he uses the term Oracle as a means of referring to a contained or "boxed" superintelligent AGI. Just because the risks of constructing an Oracle or Oracles are extremely high, it doesn't mean we should refrain from doing so. In fact, it's more safe than the alternatives: a Genie or Sovereign; superintelligences that are entirely uncontained, the latter of which operating in a hands-off fashion in accordance with its goal/value system. Given the current, seemingly intractable nature of the superintelligence control problem, one possible method in which we may accomplish safe, superintelligent AGI is to first operate many variations of potentially unsafe "Oracles" and interrogate them for information that may further our research into AGI safety, or perhaps even solve the issue entirely. Generally speaking you shouldn't trust an Oracle, but if you have many radically different designs operating in containment that are oblivious to each other, the commonality in their output when posed such questions may assist in our realization of safe AGI that can operate in an uncontained fashion.

It's more a matter of values this AI will hold than its level of intelligence. Boström argues that containment isn't really possible.

by blfr 12y ago

It's more a matter of values this AI will hold than its level of intelligence. Boström argues that containment isn't really possible.

FatalLogic 12y ago

Is there any sure way to stop the super intelligent AI from changing its own inherent values though? I guess a problem with containment is that unless it's mathematically proven to be impervious, the super intelligent AI could have figured out that it's being contained and started to learn about the real world and how to manipulate it, while the humans outside still think they're safe.

ThrustVectoring 12y ago
Changing your inherent values is a poor way to go about achieving your current set of values. The biggest issue is ensuring that everything we care about gets encapsulated as a value of the AI. If not, we wind up with things like "set off a bomb in the basement to more efficiently get people out of a burning building."
JamesArgo 12y ago
It's assumed a rational agent judges its actions by how likely said action are to align the world with their fundamental values. For this reason, it is argued, a rational agent will not change its terminal goals. For example, suppose we build an agent that desires only to make shoes. That is, its ideal universe would consist of all available matter being converted into sneakers. Would it realize this is stupid? No. It judges it actions and thoughts only by how many shoes they are likely to bring into being. Any thought against this would be judged, correctly, as a negative contribution to shoe production and so wouldn’t be implemented, and likely would remain unthought. The best paper on this by far it Basic AI Drives by Stephen M. OMOHUNDRO: http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf http://selfawaresystems.files.wordpress.com/2008/01/ai_drive... I urge you to read it. It really is a wonderful paper.
- TeMPOraL 12y ago
  The real problem is that human values are incredibly complex and messy, and any deviation from them multiplied by the power of a superintelligence would most likely doom us forever.
eli_gottlieb 12y ago
> Is there any sure way to stop the super intelligent AI from changing its own inherent values though? Yes: give it inherent values. >I guess a problem with containment is that unless it's mathematically proven to be impervious, AGI, when it eventually exists, must run according to the basic laws of learning. If you don't give it evidence that's more likely in possible-worlds with an external world than without one, it won't posit an external world. Stopping that information leak isn't that hard, provided you have full control over the agent-environment surface in the first place. It does, however, make your "agent" completely useless for anything beyond playing Atari games.

rl3 12y ago

> Boström argues that containment isn't really possible. He never argued that containment isn't possible, just that it's incredibly dangerous. In fact, in Superintelligence, he uses the term Oracle as a means of referring to a contained or "boxed" superintelligent AGI. Just because the risks of constructing an Oracle or Oracles are extremely high, it doesn't mean we should refrain from doing so. In fact, it's more safe than the alternatives: a Genie or Sovereign; superintelligences that are entirely uncontained, the latter of which operating in a hands-off fashion in accordance with its goal/value system. Given the current, seemingly intractable nature of the superintelligence control problem, one possible method in which we may accomplish safe, superintelligent AGI is to first operate many variations of potentially unsafe "Oracles" and interrogate them for information that may further our research into AGI safety, or perhaps even solve the issue entirely. Generally speaking you shouldn't trust an Oracle, but if you have many radically different designs operating in containment that are oblivious to each other, the commonality in their output when posed such questions may assist in our realization of safe AGI that can operate in an uncontained fashion.