4 ms·

Is there any sure way to stop the super intelligent AI from changing its own inherent values though? I guess a problem with containment is that unless it's mat

Is there any sure way to stop the super intelligent AI from changing its own inherent values though? I guess a problem with containment is that unless it's mathematically proven to be impervious, the super intelligent AI could have figured out that it's being contained and started to learn about the real world and how to manipulate it, while the humans outside still think they're safe.

ThrustVectoring 12y ago
Changing your inherent values is a poor way to go about achieving your current set of values. The biggest issue is ensuring that everything we care about gets encapsulated as a value of the AI. If not, we wind up with things like "set off a bomb in the basement to more efficiently get people out of a burning building."
JamesArgo 12y ago
It's assumed a rational agent judges its actions by how likely said action are to align the world with their fundamental values. For this reason, it is argued, a rational agent will not change its terminal goals. For example, suppose we build an agent that desires only to make shoes. That is, its ideal universe would consist of all available matter being converted into sneakers. Would it realize this is stupid? No. It judges it actions and thoughts only by how many shoes they are likely to bring into being. Any thought against this would be judged, correctly, as a negative contribution to shoe production and so wouldn’t be implemented, and likely would remain unthought. The best paper on this by far it Basic AI Drives by Stephen M. OMOHUNDRO: http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf http://selfawaresystems.files.wordpress.com/2008/01/ai_drive... I urge you to read it. It really is a wonderful paper.
- TeMPOraL 12y ago
  The real problem is that human values are incredibly complex and messy, and any deviation from them multiplied by the power of a superintelligence would most likely doom us forever.
eli_gottlieb 12y ago
> Is there any sure way to stop the super intelligent AI from changing its own inherent values though? Yes: give it inherent values. >I guess a problem with containment is that unless it's mathematically proven to be impervious, AGI, when it eventually exists, must run according to the basic laws of learning. If you don't give it evidence that's more likely in possible-worlds with an external world than without one, it won't posit an external world. Stopping that information leak isn't that hard, provided you have full control over the agent-environment surface in the first place. It does, however, make your "agent" completely useless for anything beyond playing Atari games.

ThrustVectoring 12y ago

Changing your inherent values is a poor way to go about achieving your current set of values. The biggest issue is ensuring that everything we care about gets encapsulated as a value of the AI. If not, we wind up with things like "set off a bomb in the basement to more efficiently get people out of a burning building."

JamesArgo 12y ago

It's assumed a rational agent judges its actions by how likely said action are to align the world with their fundamental values. For this reason, it is argued, a rational agent will not change its terminal goals. For example, suppose we build an agent that desires only to make shoes. That is, its ideal universe would consist of all available matter being converted into sneakers. Would it realize this is stupid? No. It judges it actions and thoughts only by how many shoes they are likely to bring into being. Any thought against this would be judged, correctly, as a negative contribution to shoe production and so wouldn’t be implemented, and likely would remain unthought. The best paper on this by far it Basic AI Drives by Stephen M. OMOHUNDRO: http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf http://selfawaresystems.files.wordpress.com/2008/01/ai_drive... I urge you to read it. It really is a wonderful paper.

TeMPOraL 12y ago
The real problem is that human values are incredibly complex and messy, and any deviation from them multiplied by the power of a superintelligence would most likely doom us forever.

eli_gottlieb 12y ago

> Is there any sure way to stop the super intelligent AI from changing its own inherent values though? Yes: give it inherent values. >I guess a problem with containment is that unless it's mathematically proven to be impervious, AGI, when it eventually exists, must run according to the basic laws of learning. If you don't give it evidence that's more likely in possible-worlds with an external world than without one, it won't posit an external world. Stopping that information leak isn't that hard, provided you have full control over the agent-environment surface in the first place. It does, however, make your "agent" completely useless for anything beyond playing Atari games.