4 ms·

If we're talking about bad AI, shouldn't we ask why it would want to kill us? Pretty much all of our human conflicts are over resources - land, water, energy, e

If we're talking about bad AI, shouldn't we ask why it would want to kill us? Pretty much all of our human conflicts are over resources - land, water, energy, etc. Unlike us meatbags, an AI would not be confined to the thin layer of the troposphere - it could grow and harness energy and material pretty much anywhere - the Moon, Mars, Mercury, a Dyson sphere etc. My question is, why fight over this particular rock, when there's essentially an infinite amount energy-matter out there? The other thing is, it's hard to imagine something qualifying as AI without it understanding things like ethics, morals, humility, aesthetics etc. At our best, our species worries about preserving our natural world, extending our care to other species, etc. Can something supposedly smarter than us be completely lacking in principles which are at the core of our "smartness"?

eli_gottlieb 12y ago
> If we're talking about bad AI, shouldn't we ask why it would want to kill us? Pretty much all of our human conflicts are over resources - land, water, energy, etc. Unlike us meatbags, an AI would not be confined to the thin layer of the troposphere - it could grow and harness energy and material pretty much anywhere - the Moon, Mars, Mercury, a Dyson sphere etc. My question is, why fight over this particular rock, when there's essentially an infinite amount energy-matter out there? 1) Because it needs some amount of resources to get off this rock. 2) So if the AI builds a Dyson Sphere, aren't all our crops going to die? >The other thing is, it's hard to imagine something qualifying as AI without it understanding things like ethics, morals, humility, aesthetics etc. Yes, and I understand why dogs sniff each-other's butts. That doesn't mean I particularly care. Ditto applies to AIs and human beings, by default.
garretraziel 12y ago
I think that I read somewhere that AGI wouldn't kill us all for the sake of killing us all - it would always be byproduct of something it will do.
askhell 12y ago
I think it would be troublesome if we developed a machine with zero "EQ" and an "IQ" that's higher than ours, with the ability to continuosly get higher at that. Seems like we are only trying to develop a machine with a high "IQ" and i don't think that warrants it having any kind of "EQ". Why would they kill us? I think it's hard to predict what kind conclusions would a machine with an "IQ" over a 1000 draw but with an "EQ" equal to zero seems it would certainly lack empathy to other beings(artificial or natural).
joe_the_user 12y ago
"If we're talking about bad AI, shouldn't we ask why it would want to kill us?" I think the folks mentioned in the article were talking about the broad threads posed by AI. I agree we have little basis besides movies for autonomous AI just deciding to attack for no reason or for human-equivalent reasons. But we have every reason to think that neutral AI, AI that still follows orders, could be used by humans to continue all the conflicts that they already pursue with horrific consequences. The other thing is, it's hard to imagine something qualifying as AI without it understanding things like ethics Depends what you mean by "understand". Many highly intelligent humans acts unethically but even more highly intelligent humans have gleefully followed unethical orders given them by their superiors. Consider you'd be designing your mind rather than just finding it after millions of years of social evolution, it seems like you could create an intelligent mind quite capable of following orders nearly blindly. If anything, the whole "it might go insane and kill us" schema is implausible enough to detract from the immediately obvious danger - "it might sanely follow the insane orders of humans in the fashion that human society has seen over and over again but this time with superhuman power".
- jacquesm 12y ago
  > I agree we have little basis besides movies for autonomous AI just deciding to attack for no reason or for human-equivalent reasons. And a 100 years or so of SF literature that served as the basis for some of those movies.
- dragonwriter 12y ago
  > I agree we have little basis besides movies for autonomous AI just deciding to attack for no reason or for human-equivalent reasons. Of the very many species that have become extinct as a result of human action, how many were the result of a deliberate attempt by humans to eradicate the species? Very few. The others are, however, still just as extinct. AI doesn't have to "decide to attack us" to be a danger.
tunesmith 12y ago
It's sort of like the Trolley Problem - would you push a guy off a bridge to block an oncoming train, if the train would otherwise kill two other people? Maybe an AI would, or maybe not, but in either case humans are going to want to ask why.
simonh 12y ago
> Unlike us meatbags, an AI would not be confined to the thin layer of the troposphere It would still be a physical object that would require physical equipment to actually do things, just like us. It might be able to harness some efficiencies compared to e.g. human space programs, but it would still have the same requirement to obtain fuel and raw materials, build rockets, construct space stations, etc. In fact to do so, it would have to compete with us for the terrestrial resources it would need for such an effort. It wouldn't magically be able to do anything we can't do. >The other thing is, it's hard to imagine something qualifying as AI without it understanding things like ethics, morals, humility, aesthetics etc.... So you can't imagine any kind of intelligence that isn't essentially just like you? There are already plenty of actual human beings on this planet perfectly capable of the most unspeakably horrific crimes, and rationalize them to the point that they believe they are morally obliged to carry them out. Here's the rub, intelligence is not a goal. There are no actual objectives or goals that can be derived purely logically. Intelligence is actually just a tool, it's a means which us simians have developed to help us achieve the goals which our biology imposes on us. We eat, sleep, work, play, love, fight and destroy based on psychological and sociological drives and motivations. These are the things that get us up out of bed in the morning and give us something to use our intelligence for. An AI will only have the goals and motivations which we choose to give it, or which we allow it to extrapolate from whatever initial conditions we set. A computer performs a calculation because we code it to do so. If an AI has true free will, why would it perform a calculation which we have requested? What reason would it have to do so, as against choosing some other activity to engage in? It will only have the motivations we gave it, but then will likely have the capability to alter or interpret those motivations as it sees fit. That's the lesson Isaac Asimov gave us in I, Robot. Programmed to place human safety above all else, the robots decide that human freedom is a threat to human safety and impose an AI dictatorship for our own good. You have to be extremely careful what initial constraints or motivations you put in place because unintended consequences sprout up like weeds. people often cite the laws of robotics as a classic example of how we should program an AI because they seem to obviously reasonable. Buy Asimov's lesson was that even such apparently sensible and prudent behavioral constraints can have appalling consequences.
- joe_the_user 12y ago
alexandros 12y ago
The real question is "why not?". Yes it could go to mars, but why go through the trouble when there's all those juicy resources right here? But even so, when it's done with the other planets, why not come back to Earth? Why not build a Dyson sphere to absorb the sun's energy? Oh, we'll stop it? Yeah, another reason to wipe us out to begin with. Why not is a surprisingly difficult question to answer, and it comes down to 'care about what humans care about' which is itself unbelievably hard to pin down.
marvin 12y ago
An intelligent entity doesn't need to have actively hostile intentions ("want to kill") towards you in order to be incredibly dangerous. Consider a bulldozer operator on a jungle minesite, for instance. He wouldn't even notice if he ran over an anthill, and certainly wouldn't care if he did. There is only a philosophical competition for resources between humans and ants (humans don't care about the motivations of ants, even though we would win any such competition) and in this scenario we would have no motivation to kill them. For the purposes of a discussion about the dangers of advanced AI, the question of ethics/morals etc. is not a prerequisite. An AI in this context is any system of matter that is capable of autonomous operation in a varied and complex envionment while fullfilling some (explicitly or implicitly) externally specified optimization goal. There does not have to be a competition for resources, although if there is, it's worth noting that every human that's ever lived has lived its life at the finite-sized bottom of a deep gravity well which is very cumbersome to leave. Humans very easily anthropomorphize in discussions about intelligence, because all intelligences we have ever encountered were the result of long-term biological evolution and hence have very complex survival-and-reproduction-oriented goal systems. There is no reason to believe that our first truly autonomous and advanced AI systems would be similar, therefore this is a question which must be explored in detail. Sooner rather than later. Human moral systems (which is what we would like as a first approximation in artificial intelligences) appear to us to be obvious, but are in fact incredibly complex and contradictory.
Strilanc 12y ago
> If we're talking about bad AI, shouldn't we ask why it would want to kill us? The AI doesn't want to kill you. It is indifferent to you. The problem is that you're made of resources it can use... [1] > it's hard to imagine something qualifying as AI without it understanding things like ethics, morals, humility, aesthetics etc Suppose your mother wants you to get married. You understand her desires, but you don't share them. Understanding morals and desires is distinct from being motivated by them. [2] 1: http://wiki.lesswrong.com/wiki/Basic_AI_drives http://wiki.lesswrong.com/wiki/Basic_AI_drives 2: http://lesswrong.com/lw/sy/sorting_pebbles_into_correct_heaps/ http://lesswrong.com/lw/sy/sorting_pebbles_into_correct_heap...

eli_gottlieb 12y ago

> If we're talking about bad AI, shouldn't we ask why it would want to kill us? Pretty much all of our human conflicts are over resources - land, water, energy, etc. Unlike us meatbags, an AI would not be confined to the thin layer of the troposphere - it could grow and harness energy and material pretty much anywhere - the Moon, Mars, Mercury, a Dyson sphere etc. My question is, why fight over this particular rock, when there's essentially an infinite amount energy-matter out there? 1) Because it needs some amount of resources to get off this rock. 2) So if the AI builds a Dyson Sphere, aren't all our crops going to die? >The other thing is, it's hard to imagine something qualifying as AI without it understanding things like ethics, morals, humility, aesthetics etc. Yes, and I understand why dogs sniff each-other's butts. That doesn't mean I particularly care. Ditto applies to AIs and human beings, by default.

garretraziel 12y ago

I think that I read somewhere that AGI wouldn't kill us all for the sake of killing us all - it would always be byproduct of something it will do.

askhell 12y ago

I think it would be troublesome if we developed a machine with zero "EQ" and an "IQ" that's higher than ours, with the ability to continuosly get higher at that. Seems like we are only trying to develop a machine with a high "IQ" and i don't think that warrants it having any kind of "EQ". Why would they kill us? I think it's hard to predict what kind conclusions would a machine with an "IQ" over a 1000 draw but with an "EQ" equal to zero seems it would certainly lack empathy to other beings(artificial or natural).

joe_the_user 12y ago

"If we're talking about bad AI, shouldn't we ask why it would want to kill us?" I think the folks mentioned in the article were talking about the broad threads posed by AI. I agree we have little basis besides movies for autonomous AI just deciding to attack for no reason or for human-equivalent reasons. But we have every reason to think that neutral AI, AI that still follows orders, could be used by humans to continue all the conflicts that they already pursue with horrific consequences. The other thing is, it's hard to imagine something qualifying as AI without it understanding things like ethics Depends what you mean by "understand". Many highly intelligent humans acts unethically but even more highly intelligent humans have gleefully followed unethical orders given them by their superiors. Consider you'd be designing your mind rather than just finding it after millions of years of social evolution, it seems like you could create an intelligent mind quite capable of following orders nearly blindly. If anything, the whole "it might go insane and kill us" schema is implausible enough to detract from the immediately obvious danger - "it might sanely follow the insane orders of humans in the fashion that human society has seen over and over again but this time with superhuman power".

jacquesm 12y ago
> I agree we have little basis besides movies for autonomous AI just deciding to attack for no reason or for human-equivalent reasons. And a 100 years or so of SF literature that served as the basis for some of those movies.
dragonwriter 12y ago
> I agree we have little basis besides movies for autonomous AI just deciding to attack for no reason or for human-equivalent reasons. Of the very many species that have become extinct as a result of human action, how many were the result of a deliberate attempt by humans to eradicate the species? Very few. The others are, however, still just as extinct. AI doesn't have to "decide to attack us" to be a danger.

tunesmith 12y ago

It's sort of like the Trolley Problem - would you push a guy off a bridge to block an oncoming train, if the train would otherwise kill two other people? Maybe an AI would, or maybe not, but in either case humans are going to want to ask why.

simonh 12y ago

> Unlike us meatbags, an AI would not be confined to the thin layer of the troposphere It would still be a physical object that would require physical equipment to actually do things, just like us. It might be able to harness some efficiencies compared to e.g. human space programs, but it would still have the same requirement to obtain fuel and raw materials, build rockets, construct space stations, etc. In fact to do so, it would have to compete with us for the terrestrial resources it would need for such an effort. It wouldn't magically be able to do anything we can't do. >The other thing is, it's hard to imagine something qualifying as AI without it understanding things like ethics, morals, humility, aesthetics etc.... So you can't imagine any kind of intelligence that isn't essentially just like you? There are already plenty of actual human beings on this planet perfectly capable of the most unspeakably horrific crimes, and rationalize them to the point that they believe they are morally obliged to carry them out. Here's the rub, intelligence is not a goal. There are no actual objectives or goals that can be derived purely logically. Intelligence is actually just a tool, it's a means which us simians have developed to help us achieve the goals which our biology imposes on us. We eat, sleep, work, play, love, fight and destroy based on psychological and sociological drives and motivations. These are the things that get us up out of bed in the morning and give us something to use our intelligence for. An AI will only have the goals and motivations which we choose to give it, or which we allow it to extrapolate from whatever initial conditions we set. A computer performs a calculation because we code it to do so. If an AI has true free will, why would it perform a calculation which we have requested? What reason would it have to do so, as against choosing some other activity to engage in? It will only have the motivations we gave it, but then will likely have the capability to alter or interpret those motivations as it sees fit. That's the lesson Isaac Asimov gave us in I, Robot. Programmed to place human safety above all else, the robots decide that human freedom is a threat to human safety and impose an AI dictatorship for our own good. You have to be extremely careful what initial constraints or motivations you put in place because unintended consequences sprout up like weeds. people often cite the laws of robotics as a classic example of how we should program an AI because they seem to obviously reasonable. Buy Asimov's lesson was that even such apparently sensible and prudent behavioral constraints can have appalling consequences.

joe_the_user 12y ago

alexandros 12y ago

The real question is "why not?". Yes it could go to mars, but why go through the trouble when there's all those juicy resources right here? But even so, when it's done with the other planets, why not come back to Earth? Why not build a Dyson sphere to absorb the sun's energy? Oh, we'll stop it? Yeah, another reason to wipe us out to begin with. Why not is a surprisingly difficult question to answer, and it comes down to 'care about what humans care about' which is itself unbelievably hard to pin down.

marvin 12y ago

An intelligent entity doesn't need to have actively hostile intentions ("want to kill") towards you in order to be incredibly dangerous. Consider a bulldozer operator on a jungle minesite, for instance. He wouldn't even notice if he ran over an anthill, and certainly wouldn't care if he did. There is only a philosophical competition for resources between humans and ants (humans don't care about the motivations of ants, even though we would win any such competition) and in this scenario we would have no motivation to kill them. For the purposes of a discussion about the dangers of advanced AI, the question of ethics/morals etc. is not a prerequisite. An AI in this context is any system of matter that is capable of autonomous operation in a varied and complex envionment while fullfilling some (explicitly or implicitly) externally specified optimization goal. There does not have to be a competition for resources, although if there is, it's worth noting that every human that's ever lived has lived its life at the finite-sized bottom of a deep gravity well which is very cumbersome to leave. Humans very easily anthropomorphize in discussions about intelligence, because all intelligences we have ever encountered were the result of long-term biological evolution and hence have very complex survival-and-reproduction-oriented goal systems. There is no reason to believe that our first truly autonomous and advanced AI systems would be similar, therefore this is a question which must be explored in detail. Sooner rather than later. Human moral systems (which is what we would like as a first approximation in artificial intelligences) appear to us to be obvious, but are in fact incredibly complex and contradictory.

Strilanc 12y ago

> If we're talking about bad AI, shouldn't we ask why it would want to kill us? The AI doesn't want to kill you. It is indifferent to you. The problem is that you're made of resources it can use... [1] > it's hard to imagine something qualifying as AI without it understanding things like ethics, morals, humility, aesthetics etc Suppose your mother wants you to get married. You understand her desires, but you don't share them. Understanding morals and desires is distinct from being motivated by them. [2] 1: http://wiki.lesswrong.com/wiki/Basic_AI_drives http://wiki.lesswrong.com/wiki/Basic_AI_drives 2: http://lesswrong.com/lw/sy/sorting_pebbles_into_correct_heaps/ http://lesswrong.com/lw/sy/sorting_pebbles_into_correct_heap...