6 ms·

I wonder how the goal can be achieved? How would you prevent the development of AI which is so intelligent that it is dangerous for humanity? Perhaps there mig

I wonder how the goal can be achieved? How would you prevent the development of AI which is so intelligent that it is dangerous for humanity? Perhaps there might simply be fundamental physical constraints which limit intelligence? Still seems that something much faster and bigger than a human brain would be possible, though. Maybe by containment? If it's possible to keep the super intelligent AI in a virtual environment, but it can't detect that it's virtual.

blfr 12y ago
It's more a matter of values this AI will hold than its level of intelligence. Boström argues that containment isn't really possible.
- FatalLogic 12y ago
  Is there any sure way to stop the super intelligent AI from changing its own inherent values though? I guess a problem with containment is that unless it's mathematically proven to be impervious, the super intelligent AI could have figured out that it's being contained and started to learn about the real world and how to manipulate it, while the humans outside still think they're safe.
  - ThrustVectoring 12y ago
    Changing your inherent values is a poor way to go about achieving your current set of values. The biggest issue is ensuring that everything we care about gets encapsulated as a value of the AI. If not, we wind up with things like "set off a bomb in the basement to more efficiently get people out of a burning building."
  - JamesArgo 12y ago
    It's assumed a rational agent judges its actions by how likely said action are to align the world with their fundamental values. For this reason, it is argued, a rational agent will not change its terminal goals. For example, suppose we build an agent that desires only to make shoes. That is, its ideal universe would consist of all available matter being converted into sneakers. Would it realize this is stupid? No. It judges it actions and thoughts only by how many shoes they are likely to bring into being. Any thought against this would be judged, correctly, as a negative contribution to shoe production and so wouldn’t be implemented, and likely would remain unthought. The best paper on this by far it Basic AI Drives by Stephen M. OMOHUNDRO: http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf http://selfawaresystems.files.wordpress.com/2008/01/ai_drive... I urge you to read it. It really is a wonderful paper.
    - TeMPOraL 12y ago
      The real problem is that human values are incredibly complex and messy, and any deviation from them multiplied by the power of a superintelligence would most likely doom us forever.
  - eli_gottlieb 12y ago
    > Is there any sure way to stop the super intelligent AI from changing its own inherent values though? Yes: give it inherent values. >I guess a problem with containment is that unless it's mathematically proven to be impervious, AGI, when it eventually exists, must run according to the basic laws of learning. If you don't give it evidence that's more likely in possible-worlds with an external world than without one, it won't posit an external world. Stopping that information leak isn't that hard, provided you have full control over the agent-environment surface in the first place. It does, however, make your "agent" completely useless for anything beyond playing Atari games.
- rl3 12y ago
  > Boström argues that containment isn't really possible. He never argued that containment isn't possible, just that it's incredibly dangerous. In fact, in Superintelligence, he uses the term Oracle as a means of referring to a contained or "boxed" superintelligent AGI. Just because the risks of constructing an Oracle or Oracles are extremely high, it doesn't mean we should refrain from doing so. In fact, it's more safe than the alternatives: a Genie or Sovereign; superintelligences that are entirely uncontained, the latter of which operating in a hands-off fashion in accordance with its goal/value system. Given the current, seemingly intractable nature of the superintelligence control problem, one possible method in which we may accomplish safe, superintelligent AGI is to first operate many variations of potentially unsafe "Oracles" and interrogate them for information that may further our research into AGI safety, or perhaps even solve the issue entirely. Generally speaking you shouldn't trust an Oracle, but if you have many radically different designs operating in containment that are oblivious to each other, the commonality in their output when posed such questions may assist in our realization of safe AGI that can operate in an uncontained fashion.
pavel_lishin 12y ago
I'm sorry for bringing this up, because it's vaguely controversial in some circles, and worse, a huge time-sink on par with TVTropes, but Yudkowsky attempted to run an experiment to see whether someone could be convinced by an AI, using nothing but text, to release the AI into the world. Apparently, he succeeded - his subject elected to release the fake AI. The method wasn't made public, as far as I know, and it might all be a hoax, but it's something to think about - how do you trap and imprison something smarter than you?
- ceejayoz 12y ago
  > how do you trap and imprison something smarter than you? Jails seem to have this figured out just fine.
  - JamesArgo 12y ago
    The correct analogy would be a jail run by a bacterium.
    - Retric 12y ago
      Prison bars are dumber than a bacterium and they work just fine. Still, the idea that AI is suddenly X smarter than people is ridiculously naive. Intelligence does not fit on a linear scale. And being smarter does not change the correct answer.
      - TeMPOraL 12y ago
        The problem isn't with AIs dumber than people - it's with what happens when someone finally builds a smarter one. It doesn't matter how good are prison bars if you're smart enough to social-engineer your way out of the cell. And human criminals do escape from prisons every now and then.
      - jacquesm 12y ago
        This is more like a 'reverse hacker', instead of a brilliant hacker trying to get into a system it is a brilliant hacker trying to get out of a system, and in this case the hacker is likely vastly more brilliant than the defenders. The same rules apply: the hacker has to succeed only once, the 'jailers' have to succeed all the times. Predicted long term outcome: escape.
mentos 12y ago
What about limiting AI to a certain clock rate?
- kanzure 12y ago
  > What about limiting AI to a certain clock rate? Not everyone will make high-cognitive-ability software that follows your clock rate rules. (And I think this is a good thing.)
- FatalLogic 12y ago
  Hypothetically, the AI would cunningly manufacture a crisis in which you are forced to increase its clock rate to enable it to calculate a solution to some impending disaster.
- JulianMorrison 12y ago
  http://www.rifters.com/real/2009/01/iterating-towards-bethlehem.html http://www.rifters.com/real/2009/01/iterating-towards-bethle... It's a peculiarity of humans that we can't trade off time for IQ. Do not assume an AI would be similarly limited.
- rl3 12y ago
  I'm not sure why the parent was downvoted; limiting an AI's computational capacity is considered a stunting method, and although such methods are not in and of themselves solutions to the control problem, they may be tools that assist us in developing a solution. For example, we may desire to first study AGI by restricting it to levels of intelligence slightly below that of a human. To accomplish this, we may employ various stunting and/or tripwire methods to ensure the AGI develops within our parameters.
gradys 12y ago
If homomorphic encryption gets off the ground, could an AI hide itself in homomorphically encrypted cloud compute capacity, earning Bitcoin to fund its survival?
- eli_gottlieb 12y ago
  Yes, but that relies on Bitcoin holding water over the long term. This is not so sure.
  - TeMPOraL 12y ago
    It probably wouldn't need bitcoin; it could figure out ways to trick humans into paying for the servers or just hack its way into electronic transfers.
Synthpixel 12y ago
The institute and the open letter are about maximizing the societal benefits of AI by funding research that provides net positives to humanity. The goal isn't to prevent the development of potentially harmful technologies. I believe they intend to fund positive research so that researchers do not resort to projects that do not provide net good, for example, in fields like weaponry. Not that safely developing hyperintelligent AI isn't an interesting topic. But I don't think AI research is anywhere close to that stage, considering that programmers largely still have to hard-code any actions that an agent can perform.
- butwhy 12y ago
  " The goal isn't to prevent the development of potentially harmful technologies" - actually, the goal includes "avoiding potential pitfalls". One would assume that creating danger towards humanity would be a pitfall.
yoha 12y ago
About the idea of containing an AI → https://en.wikipedia.org/wiki/AI_box_experiment https://en.wikipedia.org/wiki/AI_box_experiment
evanspa 12y ago
"If it's possible to keep the super intelligent AI in a virtual environment, but it can't detect that it's virtual." They did this in a Star Trek TNG episode ("Ship in a Bottle"). Data accidentally endows a holodeck character with super-intelligence. The way the crew eventually defeated him was to trick him into thinking he had left the holodeck and was able to live normally in the outside world; but it was really just another virtual environment.
- pzxc 12y ago
  Actually, Geordi endows Professor Moriarty with super-intelligence, by mistakenly asking the holodeck/computer to create an adversary "capable of defeating Data" instead of "capable of defeating Holmes" as he intended. God, I'm such a nerd. :P
  - colordrops 12y ago
    So the holodeck computer must be insanely intelligent. Or, it turns out that the building blocks for intelligence are simple and deterministic.
  - deleted 12y ago
    [deleted]
  - evanspa 12y ago
    Yup. Good catch :)
- comrade1 12y ago
  He doesn't sound that smart...
Retra 12y ago
Intelligent doesn't imply dangerous. It doesn't imply _anything_ so long as it is underspecified. That is the core problem: what do we actually want, and how do we know it exists? We want something that thinks enough like a human to understand and appreciate our goals, but enough different from a human to not suffer our faults. And in asking for this, we already suppose the existence of an AI that fundamentally disagrees with how humans make decisions. This is the 'better morality' problem. If I say "I'm more moral than you", I have to do so by showing that _your_ morality is contradictory with _your _ actions; that you are a hypocrite. Otherwise, we will disagree on some course of action, and you will have no reason to believe your decisions are incorrect except by _assuming_ that I am more moral, which is what we are trying to establish. So we really want a computer that can point out our errors and suggest corrections. This is one (vague) foundation for artificial intelligence that doesn't easily permit 'evil geniuses' since, so long as this is what the computer is doing, it cannot be said to be dangerous. Overall, I find AI discussion to be incredibly primitive and naive. It's practically a religious debate except among a very very few. Most people are not very creative about addressing these kinds of issues, and it's depressing that people are tied up in so thoroughly in bad terminology and non-issues.
- ThomPete 12y ago
  It doesn't have to have intent to be dangerous thats the problem. Just like you don't mean to step on an ant but probably have often done that.
  - Retra 12y ago
    It is far harder to step on ants when you are designed to avoid stepping on them. Which naturally, those ants will try to do.
    - ThomPete 12y ago
      Which require the ants understand what kind of "steps" it can take which they won't.
      - 12y ago
alexvr 12y ago
My first instinct is to put it in a virtual world, but if it's smart enough, it could perhaps quickly realize the nature of its world and feign ignorance. Then, under the pretext of simply improving its own quality of life, it could tempt us by engineering awesome things that would help us tremendously in the real world -- all we would have to do is copy its work (because we wouldn't be able to fully comprehend it), and our energy problems become a thing of the past! But the AI forgot to mention that its wonderful creation also gives it access to our physical world and puts us at its mercy.

blfr 12y ago

It's more a matter of values this AI will hold than its level of intelligence. Boström argues that containment isn't really possible.

FatalLogic 12y ago
Is there any sure way to stop the super intelligent AI from changing its own inherent values though? I guess a problem with containment is that unless it's mathematically proven to be impervious, the super intelligent AI could have figured out that it's being contained and started to learn about the real world and how to manipulate it, while the humans outside still think they're safe.
- ThrustVectoring 12y ago
  Changing your inherent values is a poor way to go about achieving your current set of values. The biggest issue is ensuring that everything we care about gets encapsulated as a value of the AI. If not, we wind up with things like "set off a bomb in the basement to more efficiently get people out of a burning building."
- JamesArgo 12y ago
  It's assumed a rational agent judges its actions by how likely said action are to align the world with their fundamental values. For this reason, it is argued, a rational agent will not change its terminal goals. For example, suppose we build an agent that desires only to make shoes. That is, its ideal universe would consist of all available matter being converted into sneakers. Would it realize this is stupid? No. It judges it actions and thoughts only by how many shoes they are likely to bring into being. Any thought against this would be judged, correctly, as a negative contribution to shoe production and so wouldn’t be implemented, and likely would remain unthought. The best paper on this by far it Basic AI Drives by Stephen M. OMOHUNDRO: http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf http://selfawaresystems.files.wordpress.com/2008/01/ai_drive... I urge you to read it. It really is a wonderful paper.
  - TeMPOraL 12y ago
    The real problem is that human values are incredibly complex and messy, and any deviation from them multiplied by the power of a superintelligence would most likely doom us forever.
- eli_gottlieb 12y ago
  > Is there any sure way to stop the super intelligent AI from changing its own inherent values though? Yes: give it inherent values. >I guess a problem with containment is that unless it's mathematically proven to be impervious, AGI, when it eventually exists, must run according to the basic laws of learning. If you don't give it evidence that's more likely in possible-worlds with an external world than without one, it won't posit an external world. Stopping that information leak isn't that hard, provided you have full control over the agent-environment surface in the first place. It does, however, make your "agent" completely useless for anything beyond playing Atari games.
rl3 12y ago
> Boström argues that containment isn't really possible. He never argued that containment isn't possible, just that it's incredibly dangerous. In fact, in Superintelligence, he uses the term Oracle as a means of referring to a contained or "boxed" superintelligent AGI. Just because the risks of constructing an Oracle or Oracles are extremely high, it doesn't mean we should refrain from doing so. In fact, it's more safe than the alternatives: a Genie or Sovereign; superintelligences that are entirely uncontained, the latter of which operating in a hands-off fashion in accordance with its goal/value system. Given the current, seemingly intractable nature of the superintelligence control problem, one possible method in which we may accomplish safe, superintelligent AGI is to first operate many variations of potentially unsafe "Oracles" and interrogate them for information that may further our research into AGI safety, or perhaps even solve the issue entirely. Generally speaking you shouldn't trust an Oracle, but if you have many radically different designs operating in containment that are oblivious to each other, the commonality in their output when posed such questions may assist in our realization of safe AGI that can operate in an uncontained fashion.

pavel_lishin 12y ago

I'm sorry for bringing this up, because it's vaguely controversial in some circles, and worse, a huge time-sink on par with TVTropes, but Yudkowsky attempted to run an experiment to see whether someone could be convinced by an AI, using nothing but text, to release the AI into the world. Apparently, he succeeded - his subject elected to release the fake AI. The method wasn't made public, as far as I know, and it might all be a hoax, but it's something to think about - how do you trap and imprison something smarter than you?

ceejayoz 12y ago
> how do you trap and imprison something smarter than you? Jails seem to have this figured out just fine.
- JamesArgo 12y ago
  The correct analogy would be a jail run by a bacterium.
  - Retric 12y ago
    Prison bars are dumber than a bacterium and they work just fine. Still, the idea that AI is suddenly X smarter than people is ridiculously naive. Intelligence does not fit on a linear scale. And being smarter does not change the correct answer.
    - TeMPOraL 12y ago
      The problem isn't with AIs dumber than people - it's with what happens when someone finally builds a smarter one. It doesn't matter how good are prison bars if you're smart enough to social-engineer your way out of the cell. And human criminals do escape from prisons every now and then.
    - jacquesm 12y ago
      This is more like a 'reverse hacker', instead of a brilliant hacker trying to get into a system it is a brilliant hacker trying to get out of a system, and in this case the hacker is likely vastly more brilliant than the defenders. The same rules apply: the hacker has to succeed only once, the 'jailers' have to succeed all the times. Predicted long term outcome: escape.

mentos 12y ago

What about limiting AI to a certain clock rate?

kanzure 12y ago
> What about limiting AI to a certain clock rate? Not everyone will make high-cognitive-ability software that follows your clock rate rules. (And I think this is a good thing.)
FatalLogic 12y ago
Hypothetically, the AI would cunningly manufacture a crisis in which you are forced to increase its clock rate to enable it to calculate a solution to some impending disaster.
JulianMorrison 12y ago
http://www.rifters.com/real/2009/01/iterating-towards-bethlehem.html http://www.rifters.com/real/2009/01/iterating-towards-bethle... It's a peculiarity of humans that we can't trade off time for IQ. Do not assume an AI would be similarly limited.
rl3 12y ago
I'm not sure why the parent was downvoted; limiting an AI's computational capacity is considered a stunting method, and although such methods are not in and of themselves solutions to the control problem, they may be tools that assist us in developing a solution. For example, we may desire to first study AGI by restricting it to levels of intelligence slightly below that of a human. To accomplish this, we may employ various stunting and/or tripwire methods to ensure the AGI develops within our parameters.

gradys 12y ago

If homomorphic encryption gets off the ground, could an AI hide itself in homomorphically encrypted cloud compute capacity, earning Bitcoin to fund its survival?

eli_gottlieb 12y ago
Yes, but that relies on Bitcoin holding water over the long term. This is not so sure.
- TeMPOraL 12y ago
  It probably wouldn't need bitcoin; it could figure out ways to trick humans into paying for the servers or just hack its way into electronic transfers.

Synthpixel 12y ago

The institute and the open letter are about maximizing the societal benefits of AI by funding research that provides net positives to humanity. The goal isn't to prevent the development of potentially harmful technologies. I believe they intend to fund positive research so that researchers do not resort to projects that do not provide net good, for example, in fields like weaponry. Not that safely developing hyperintelligent AI isn't an interesting topic. But I don't think AI research is anywhere close to that stage, considering that programmers largely still have to hard-code any actions that an agent can perform.

butwhy 12y ago
" The goal isn't to prevent the development of potentially harmful technologies" - actually, the goal includes "avoiding potential pitfalls". One would assume that creating danger towards humanity would be a pitfall.

yoha 12y ago

About the idea of containing an AI → https://en.wikipedia.org/wiki/AI_box_experiment https://en.wikipedia.org/wiki/AI_box_experiment

evanspa 12y ago

"If it's possible to keep the super intelligent AI in a virtual environment, but it can't detect that it's virtual." They did this in a Star Trek TNG episode ("Ship in a Bottle"). Data accidentally endows a holodeck character with super-intelligence. The way the crew eventually defeated him was to trick him into thinking he had left the holodeck and was able to live normally in the outside world; but it was really just another virtual environment.

pzxc 12y ago
Actually, Geordi endows Professor Moriarty with super-intelligence, by mistakenly asking the holodeck/computer to create an adversary "capable of defeating Data" instead of "capable of defeating Holmes" as he intended. God, I'm such a nerd. :P
- colordrops 12y ago
  So the holodeck computer must be insanely intelligent. Or, it turns out that the building blocks for intelligence are simple and deterministic.
- deleted 12y ago
  [deleted]
- evanspa 12y ago
  Yup. Good catch :)
comrade1 12y ago
He doesn't sound that smart...

Retra 12y ago

Intelligent doesn't imply dangerous. It doesn't imply _anything_ so long as it is underspecified. That is the core problem: what do we actually want, and how do we know it exists? We want something that thinks enough like a human to understand and appreciate our goals, but enough different from a human to not suffer our faults. And in asking for this, we already suppose the existence of an AI that fundamentally disagrees with how humans make decisions. This is the 'better morality' problem. If I say "I'm more moral than you", I have to do so by showing that _your_ morality is contradictory with _your _ actions; that you are a hypocrite. Otherwise, we will disagree on some course of action, and you will have no reason to believe your decisions are incorrect except by _assuming_ that I am more moral, which is what we are trying to establish. So we really want a computer that can point out our errors and suggest corrections. This is one (vague) foundation for artificial intelligence that doesn't easily permit 'evil geniuses' since, so long as this is what the computer is doing, it cannot be said to be dangerous. Overall, I find AI discussion to be incredibly primitive and naive. It's practically a religious debate except among a very very few. Most people are not very creative about addressing these kinds of issues, and it's depressing that people are tied up in so thoroughly in bad terminology and non-issues.

ThomPete 12y ago
It doesn't have to have intent to be dangerous thats the problem. Just like you don't mean to step on an ant but probably have often done that.
- Retra 12y ago
  It is far harder to step on ants when you are designed to avoid stepping on them. Which naturally, those ants will try to do.
  - ThomPete 12y ago
    Which require the ants understand what kind of "steps" it can take which they won't.
    - 12y ago

alexvr 12y ago

My first instinct is to put it in a virtual world, but if it's smart enough, it could perhaps quickly realize the nature of its world and feign ignorance. Then, under the pretext of simply improving its own quality of life, it could tempt us by engineering awesome things that would help us tremendously in the real world -- all we would have to do is copy its work (because we wouldn't be able to fully comprehend it), and our energy problems become a thing of the past! But the AI forgot to mention that its wonderful creation also gives it access to our physical world and puts us at its mercy.