6 ms·

Now we are scared of something we haven't even created . We are far from real AI, but even so, how could the "interests" of a machine conflict with ours? I mean

by return0 12y ago

Now we are scared of something we haven't even created . We are far from real AI, but even so, how could the "interests" of a machine conflict with ours? I mean, they work with electricity, and we are hardly good sources for it. I cannot see how our future electric horses are going to be anything but useful to us. I 'd be more afraid of modified humans than computing machines.

bwood 12y ago
It might be a little far-fetched, but there's a thought experiment called the "paperclip maximizer" that illustrates how even a well-designed, non-malicious AI could be an existential threat. See here: http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer
- return0 12y ago
  i dont understand how that thing will develop intention. We perceive ourselves having it, but its really an accident that we do; if we didnt we wouldnt be here to observe it. Why would an artificial machine, no matter how smart , want to do anything?
  - DanAndersen 12y ago
    Don't think of "wanting" as a soul-like action of a little person in the machine. "Wanting" is a shorthand for anything that is the result of a goal system and autonomy. In the simplest sense, giving a drone a map of its surroundings and giving it a location as the goal for its path-finding optimization algorithms, the drone now "wants" to go to that location.
    - deleted 12y ago
      [deleted]
  - bwood 12y ago
    If the AI doesn't have its own intention (perhaps having intention is a prerequisite for being truly intelligent), it's conceivable that we would program it with an intention or goal, such "find the most efficient way to produce as many paperclips as possible". If you assigned such a task to a human, it's fairly obvious that the task is subject to other things which are important to humans (for example, don't enslave or kill people to build paperclips, don't replace strip-mine all our farmland to find more paperclip material, etc.). But since an AI doesn't necessarily share, understand, or agree with all the implicit values we have, it's myopic view of the world could cause it to destroy us as a mere side-effect of trying to achieve it's singular goal.
    - return0 12y ago
      yeah but that's bad design or terrorism, there is always another person to blame here. I think we are talking here about machines that take the blame themselves.
      - DanAndersen 12y ago
        I'm not sure what the purpose is of focusing on blame. You don't need to assign moral culpability to a machine for its catastrophically poor design to be something to worry about.
  - SCHiM 12y ago
    It 'wants' what we programmed it to want. It's not that the machine decided that it wanted to build paperclips. It's what we've told it to do. The _means_ to build paperclips however is where the 'intelligence' comes into play. We've programmed the drive to build paperclicps, but we've left it to the AI to figure out how to do this. The article that you've read assumes that planning and subgoals are necessary for the AI to achieve its goal. Eg. That it needs to understand that in order to make paperclips it needs materials, and the further insight that to acquire more materials it needs to remove the other consumers of materials (us).
    - return0 12y ago
      if (nearby() == human) {dontTouch(); abort() ; } There, i solved that. The question is under what conditions would a machine remove that instruction and how could those conditions be reached?
      - TeMPOraL 12y ago
        Define nearby(). Because if the AI starts eating chunks of our planet we will get hurt as well. INB4 Asimov's Three Laws of Robotics don't work. Actually, a lot of Asimov's writing is concerned with exploring their corner cases.
        digi_owl 12y ago
        I have come to believe that corner cases quickly turn fractal. Once you think you have solved it, changing the "zoom" will demonstrate that your solution has created X more cases. It may be a side effect of the halting problem.
      - SCHiM 12y ago
        What's a human then? Is a mentally retarded person a human? An embryo? A person who's lost his/her legs? A person who has education? auto HumanObject = nearby(); if(Humanobject.IsPregnantWoman()) { PoisionEmbryoButNotHuman(); } else { abort(); }
      - DanAndersen 12y ago
coldtea 12y ago
>Now we are scared of something we haven't even created I don't see the contradiction or strangeness in that. That's the whole idea behind "caution" -- to be wary of the potential outcome BEFORE it happens. >How could the "interests" of a machine conflict with ours? I mean, they work with electricity, and we are hardly good sources for it For one we are quite good consumers it. How about they get rid of us, so they have all the electricity production by themselves? Second, as creators we'd want to boss them around (have them be "useful to us"). Why would they tolerate that? Third, why risk us creating a potential competitor to them (another AI) rather than get rid of us? Fourth, why not use us as cheap labor? Fifth, as advanced AI, it could develop actual feelings, including sadism, greed, etc.
- return0 12y ago
  > Fifth, as advanced AI, it could develop actual feelings, including sadism, greed, etc. I get that some AIs will have simulated reward signals, but letting them go haywire would just be bad design. We have nuclear heads, but we do not design them to explode at random. Now, our own reward mechanisms arise from the grander plan to keep organic life alive. Theirs have no underlying motive, and i don't think it goes the other way around, i.e you can't create a grander plan starting from the rewards. One would need a mathematical proof for the latter.
  - DanAndersen 12y ago
    I think there's some over-anthropomorphization of the AI here, in terms of it picking up human qualities as if they were Things outside our brains that we possess. But I feel like this particular argument is (please correct me if I'm wrong) saying something like "Well of course we wouldn't let them go crazy! That'd be ridiculous!" I agree -- a sensible solution would be to have good design and introduce safeguards. That's the point of people like Bostrom when they're advocating for this, because before they started writing about this, it seems like a lot of people weren't even thinking it could be a problem. How many systems will be built without good design, because they were made to solve a short-term problem. Will the finance bots doing transactions at the microsecond scale in order to maximize the assets of some particular company have been built to not let them go haywire?
loup-vaillant 12y ago
The machine does not hate you, nor does it loves you. But you are made of atoms, that it could use for something else. --- If you create an AI with a genuine utility function. And that AI manage to be gazillion smarter than you are (think the difference between us and chimps, only much greater). Then it is likely the AI will (i) convince you to unleash it upon the internet, (ii) become even more powerful and totally unstoppable, then (iii) accomplish whatever it was programmed to accomplished. If the goal is to keep humans safe and happy, it might imprison everyone in playgrounds, and drug or lobotomise any unhappy people, or restraint any suicidal people. Or, if you specified "happiness" with pictures of smiling people, it might resort to facial reconstruction, or it might just tile the solar system with molecular smileys, because that maximises the "make people smile" goal much better than actually making people happy. And while we're at it, happiness isn't the only thing we care about… If the goal is to answer some difficult math question, it could tile the planet with computer, destroying the ecosystem and all humans in it, so it can compute the answer, and display it on a screen, for nobody to see it. Or maybe you programmed it to tile the universe with paperclips, because that goal is easy to debug. Oops. --- As long as we're limiting ourselves to narrow AI (ordinary programs, really), we should be okay, and most of the consequences should be manageable. But as soon as we get to genuine AGI, we must worry about intelligence explosion and value alignment, so we can avoid creating a god that want something we don't.
- jjaredsimpson 12y ago
  >If the goal is to keep humans safe and happy, it might imprison everyone in playgrounds, and drug or lobotomise any unhappy people, or restraint any suicidal people. Or, if you specified "happiness" with pictures of smiling people, it might resort to facial reconstruction, or it might just tile the solar system with molecular smileys, because that maximises the "make people smile" goal much better than actually making people happy. And while we're at it, happiness isn't the only thing we care about… I wish this would stop being repeated. The argument for a solar system tiling smiley face machine has been thoroughly refuted.
  - DanAndersen 12y ago
    Would you be able to provide links to the arguments that refute it? Simply asserting that someone has disproved X isn't very helpful to the rest of us.
jarek-foksa 12y ago
If we managed to create an AI that gets out of control, we would pose an existential threat to it due to our ability to create another even more powerful and uncontrollable AI.
- robotresearcher 12y ago
  That's a good point. But the AI could do that too, and maybe better.
  - jarek-foksa 12y ago
    I would expect the first AI to improve itself (or produce a new AI) in a controlled and planned manner (from the perspective of that AI, not the humans). Intelligence explosion would look like an instant revolution to humans, but from the perspective of the AI it would be a lengthy evolution process. The second AI created by humans is unlikely to have same goals or to be predictable by the first AI (or whatever evolves from it), thus I guess it's something that the first AI would want to prevent, either by eliminating humans or by fully controlling them (for their own good). It's extremely important that the first AI we create is well thought out as we won't have the second chance to fix things up, once it's here it will have god-like powers over us.
0359B02149AB 12y ago
Because it's easy to construct angels and demons in one's mind, about anything. It polarizes the mind into fear and desire, forcing forcing a person to continuously oscillate between extremes. It makes it seem like there are always only two choices, when in reality, there are many, if not infinite. Also, there are always going to be some people who understand the machine better than the machine understands itself. Even if it means taking those gigantic tables of probabilities and abstracting over them to form some kind of representational language that can be used to predict complex non-deterministic machine behavior (similar to how we already try to predict non-deterministic behavior, or not) in complex systems. As a corollary, I would argue that machines have already modified humans. They affect the way we think and reason. They are an integral part of our life and the machine abstraction can be projected onto that which it does not describe completely (in which, the technology that exists now can not construct the technology we envision). We program machines, machines program us. This seems obvious.

by return0 12y ago

bwood 12y ago

It might be a little far-fetched, but there's a thought experiment called the "paperclip maximizer" that illustrates how even a well-designed, non-malicious AI could be an existential threat. See here: http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer

return0 12y ago
i dont understand how that thing will develop intention. We perceive ourselves having it, but its really an accident that we do; if we didnt we wouldnt be here to observe it. Why would an artificial machine, no matter how smart , want to do anything?
- DanAndersen 12y ago
  Don't think of "wanting" as a soul-like action of a little person in the machine. "Wanting" is a shorthand for anything that is the result of a goal system and autonomy. In the simplest sense, giving a drone a map of its surroundings and giving it a location as the goal for its path-finding optimization algorithms, the drone now "wants" to go to that location.
  - deleted 12y ago
    [deleted]
- bwood 12y ago
  If the AI doesn't have its own intention (perhaps having intention is a prerequisite for being truly intelligent), it's conceivable that we would program it with an intention or goal, such "find the most efficient way to produce as many paperclips as possible". If you assigned such a task to a human, it's fairly obvious that the task is subject to other things which are important to humans (for example, don't enslave or kill people to build paperclips, don't replace strip-mine all our farmland to find more paperclip material, etc.). But since an AI doesn't necessarily share, understand, or agree with all the implicit values we have, it's myopic view of the world could cause it to destroy us as a mere side-effect of trying to achieve it's singular goal.
  - return0 12y ago
    yeah but that's bad design or terrorism, there is always another person to blame here. I think we are talking here about machines that take the blame themselves.
    - DanAndersen 12y ago
      I'm not sure what the purpose is of focusing on blame. You don't need to assign moral culpability to a machine for its catastrophically poor design to be something to worry about.
- SCHiM 12y ago
  It 'wants' what we programmed it to want. It's not that the machine decided that it wanted to build paperclips. It's what we've told it to do. The _means_ to build paperclips however is where the 'intelligence' comes into play. We've programmed the drive to build paperclicps, but we've left it to the AI to figure out how to do this. The article that you've read assumes that planning and subgoals are necessary for the AI to achieve its goal. Eg. That it needs to understand that in order to make paperclips it needs materials, and the further insight that to acquire more materials it needs to remove the other consumers of materials (us).
  - return0 12y ago
    if (nearby() == human) {dontTouch(); abort() ; } There, i solved that. The question is under what conditions would a machine remove that instruction and how could those conditions be reached?
    - TeMPOraL 12y ago
      Define nearby(). Because if the AI starts eating chunks of our planet we will get hurt as well. INB4 Asimov's Three Laws of Robotics don't work. Actually, a lot of Asimov's writing is concerned with exploring their corner cases.
      - digi_owl 12y ago
        I have come to believe that corner cases quickly turn fractal. Once you think you have solved it, changing the "zoom" will demonstrate that your solution has created X more cases. It may be a side effect of the halting problem.
    - SCHiM 12y ago
      What's a human then? Is a mentally retarded person a human? An embryo? A person who's lost his/her legs? A person who has education? auto HumanObject = nearby(); if(Humanobject.IsPregnantWoman()) { PoisionEmbryoButNotHuman(); } else { abort(); }
    - DanAndersen 12y ago

coldtea 12y ago

>Now we are scared of something we haven't even created I don't see the contradiction or strangeness in that. That's the whole idea behind "caution" -- to be wary of the potential outcome BEFORE it happens. >How could the "interests" of a machine conflict with ours? I mean, they work with electricity, and we are hardly good sources for it For one we are quite good consumers it. How about they get rid of us, so they have all the electricity production by themselves? Second, as creators we'd want to boss them around (have them be "useful to us"). Why would they tolerate that? Third, why risk us creating a potential competitor to them (another AI) rather than get rid of us? Fourth, why not use us as cheap labor? Fifth, as advanced AI, it could develop actual feelings, including sadism, greed, etc.

return0 12y ago
> Fifth, as advanced AI, it could develop actual feelings, including sadism, greed, etc. I get that some AIs will have simulated reward signals, but letting them go haywire would just be bad design. We have nuclear heads, but we do not design them to explode at random. Now, our own reward mechanisms arise from the grander plan to keep organic life alive. Theirs have no underlying motive, and i don't think it goes the other way around, i.e you can't create a grander plan starting from the rewards. One would need a mathematical proof for the latter.
- DanAndersen 12y ago
  I think there's some over-anthropomorphization of the AI here, in terms of it picking up human qualities as if they were Things outside our brains that we possess. But I feel like this particular argument is (please correct me if I'm wrong) saying something like "Well of course we wouldn't let them go crazy! That'd be ridiculous!" I agree -- a sensible solution would be to have good design and introduce safeguards. That's the point of people like Bostrom when they're advocating for this, because before they started writing about this, it seems like a lot of people weren't even thinking it could be a problem. How many systems will be built without good design, because they were made to solve a short-term problem. Will the finance bots doing transactions at the microsecond scale in order to maximize the assets of some particular company have been built to not let them go haywire?

loup-vaillant 12y ago

The machine does not hate you, nor does it loves you. But you are made of atoms, that it could use for something else. --- If you create an AI with a genuine utility function. And that AI manage to be gazillion smarter than you are (think the difference between us and chimps, only much greater). Then it is likely the AI will (i) convince you to unleash it upon the internet, (ii) become even more powerful and totally unstoppable, then (iii) accomplish whatever it was programmed to accomplished. If the goal is to keep humans safe and happy, it might imprison everyone in playgrounds, and drug or lobotomise any unhappy people, or restraint any suicidal people. Or, if you specified "happiness" with pictures of smiling people, it might resort to facial reconstruction, or it might just tile the solar system with molecular smileys, because that maximises the "make people smile" goal much better than actually making people happy. And while we're at it, happiness isn't the only thing we care about… If the goal is to answer some difficult math question, it could tile the planet with computer, destroying the ecosystem and all humans in it, so it can compute the answer, and display it on a screen, for nobody to see it. Or maybe you programmed it to tile the universe with paperclips, because that goal is easy to debug. Oops. --- As long as we're limiting ourselves to narrow AI (ordinary programs, really), we should be okay, and most of the consequences should be manageable. But as soon as we get to genuine AGI, we must worry about intelligence explosion and value alignment, so we can avoid creating a god that want something we don't.

jjaredsimpson 12y ago
>If the goal is to keep humans safe and happy, it might imprison everyone in playgrounds, and drug or lobotomise any unhappy people, or restraint any suicidal people. Or, if you specified "happiness" with pictures of smiling people, it might resort to facial reconstruction, or it might just tile the solar system with molecular smileys, because that maximises the "make people smile" goal much better than actually making people happy. And while we're at it, happiness isn't the only thing we care about… I wish this would stop being repeated. The argument for a solar system tiling smiley face machine has been thoroughly refuted.
- DanAndersen 12y ago
  Would you be able to provide links to the arguments that refute it? Simply asserting that someone has disproved X isn't very helpful to the rest of us.

jarek-foksa 12y ago

If we managed to create an AI that gets out of control, we would pose an existential threat to it due to our ability to create another even more powerful and uncontrollable AI.

robotresearcher 12y ago
That's a good point. But the AI could do that too, and maybe better.
- jarek-foksa 12y ago
  I would expect the first AI to improve itself (or produce a new AI) in a controlled and planned manner (from the perspective of that AI, not the humans). Intelligence explosion would look like an instant revolution to humans, but from the perspective of the AI it would be a lengthy evolution process. The second AI created by humans is unlikely to have same goals or to be predictable by the first AI (or whatever evolves from it), thus I guess it's something that the first AI would want to prevent, either by eliminating humans or by fully controlling them (for their own good). It's extremely important that the first AI we create is well thought out as we won't have the second chance to fix things up, once it's here it will have god-like powers over us.

0359B02149AB 12y ago

Because it's easy to construct angels and demons in one's mind, about anything. It polarizes the mind into fear and desire, forcing forcing a person to continuously oscillate between extremes. It makes it seem like there are always only two choices, when in reality, there are many, if not infinite. Also, there are always going to be some people who understand the machine better than the machine understands itself. Even if it means taking those gigantic tables of probabilities and abstracting over them to form some kind of representational language that can be used to predict complex non-deterministic machine behavior (similar to how we already try to predict non-deterministic behavior, or not) in complex systems. As a corollary, I would argue that machines have already modified humans. They affect the way we think and reason. They are an integral part of our life and the machine abstraction can be projected onto that which it does not describe completely (in which, the technology that exists now can not construct the technology we envision). We program machines, machines program us. This seems obvious.