8 ms·

If we create machines that learn as well as our brains do, it’s easy to imagine them inheriting human-like qualities—and flaws. But a “Terminator”-style scenari

If we create machines that learn as well as our brains do, it’s easy to imagine them inheriting human-like qualities—and flaws. But a “Terminator”-style scenario is, in my view, immensely improbable. It would require a discrete, malevolent entity to specifically hard-wire malicious intent into intelligent machines, and no organization, let alone a single group or a person, will achieve human-level AI alone. Isn't this incredibly shortsighted? Ignoring all the questions regarding the morals and ethics an intelligent machine may feel and affect the way it behaves... It used to take nations to build computers, then large corporations, then off-the-shelf parts by a kid in his garage. The first strong AI will most likely be a multi-billion dollar project, but its creation will arguably usher in an era in which strong AI is ubiquitous.

saiya-jin 10y ago
it would be perfectly OK if we would be living in peaceful world without any conflict, without nations which should hold the banner of democracy and freedom, where there are no terrorist attacks and so on. last time i checked, we're getting further and further away from this scenario. in fact, we SHOULD expect evil AI in worst form possible and beyond, anything else is dangerously naive.
- fennecfoxen 10y ago
  We should expect evil AI. But it's not patently obvious that we should expect it to make the world worse off than evil natural intelligences. The horrors of World War II and the Holocaust were accomplished using humans, and global thermonuclear annihilation was on the table while computers were still programmed with core rope memory strung together by little old ladies with knitting skills. Perhaps there is something specific about the availability of artificial intelligences, and their relation to the world around them, that is different and intrinsically more threatening than a world containing only human intelligences - but surely we mustn't leap to that conclusion without identifying what this difference is.
  - ZanyProgrammer 10y ago
    "We should expect evil AI."-with your certainty, I'd appreciate some lottery numbers.
    - fennecfoxen 10y ago
      Honestly, if nothing else, someone will make it their pet project as a sick joke. Of course, that doesn't mean there'll necessarily be a high-quality or impactful evil AI, mind you... but barring global thermonuclear annihilation it will happen sooner or later. Your lottery numbers are 23skidoo. Your I Ching reading is ䷍ (Wealth, great treasures, supreme success).
fennecfoxen 10y ago
The Terminator scenario is basically one, single centralized AI wired into everything (or at least wired into enough military power to stave off opposition as it force-wires itself into all the rest of the things). But practically speaking, a single centralized AI is highly unlikely to be a world-takeover machine of godlike intelligence in its first iteration. The iterative improvement that you refer to which could potentially make it dangerous also points to a likelihood of there being a diversity of agents with a variety of designs being run by a diverse set of organizations with a diverse set of goals, with many incentives to obtain protection from hostile AIs and computer systems run by 1337 h4x0rz (possibly foreign government hackers). Diversity and independence of AI Agents mitigates against the Terminator-scenario danger; the situation would really be not so different than the modern-day situation with natural intelligences that have control over resources and weapons of war.
- gm-conspiracy 10y ago
  I am not so worried about explicit physical violence, but subtle manipulation of what us humans consider reality. Also, are you assuming the iterative development requires human intervention?
  - fennecfoxen 10y ago
    > Also, are you assuming the iterative development requires human intervention? At first, yes; long-term, it doesn't matter too much. If IBM's AI agents come up with better versions of themselves they'll still be trying to rent them out to different people in order to compete with Google's and Baidu's and Lockheed's and Siemens' businesses in those market (or whoever's still around and actually running these things at that point).
    - gm-conspiracy 10y ago
      So, multiple benevolent feral AIs that coexist in harmony?
  - TheOtherHobbes 10y ago
emiliobumachar 10y ago
No malice is necessary. https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."
- fratlas 10y ago
  I believe this is the most likely scenario: lethal apathy. I mean, how could a turing machine have emotions without chemicals?
  - drabiega 10y ago
    Emotions are a heuristic. Any general AI powerful enough to be useful is going to have to be full of similar heuristics if it's going to get anything done. The presence of chemicals is irrelevant.
    - deleted 10y ago
      [deleted]
    - maroonblazer 10y ago
      In what way are emotions a dependency on getting anything done? The AI could be like the high-functioning sociopath who doesn't actually experience the subjective feeling of emotion but has learned all the physical cues that convey a given emotion. An AI would execute them flawlessly.
  - adrianN 10y ago
    What have emotions to do with this? And what have chemicals to do with emotions?
- wapapaloobop 10y ago
  The idea that I'll use human bodies if I need a bunch of atoms makes as much sense as the premise of The Matrix i.e. harvesting humans for energy production.
snemvalts 10y ago
Andrew Ng made a really good analogy to those afraid of strong AI destroying humanity: "It's like being afraid of overpopulation on Mars, we haven't even landed on the planet yet."
- FeepingCreature 10y ago
  To be fair, we do worry about contamination of Mars with microorganisms, which I believe is a better analogue for something with a potential exponential takeoff.
- ehudla 10y ago
  It's more like we haven't discovered Mars yet.
- Houshalter 10y ago
  It's a stupid analogy. Mars overpopulation would obviously take many, many centuries. It would be a slow thing that you could obviously see coming. There is no reason to believe AI will take centuries to build, or that we will necessarily see it coming. A better example might be like H.G. Well's 1913 prediction of nuclear weapons destroying the world. It was something that science was just realizing was possible, and would be invented within his lifetime.
  - snemvalts 10y ago
    We're far from emulating networks on the scale of the visual cortex, let alone a self-reasoning machine (we don't even fully understand consciousness and inner workings of the brain). People fearing strong-AI are the ones not involved in the field, yet all this hype/fear from them (in combination with Moore's law ending) is probably going to cause another AI winter.
    - Houshalter 10y ago
      And in 1913 we didn't have even basic nuclear technology. Just 3 decades is a long time for newly emerging technologies. >We're far from emulating networks on the scale of the visual cortex In 2009 (ish) computer vision was a joke that could recognize very few objects a small percent of the time. Based on only simple color and texture, and sometimes basic shapes. A few years later and computers were excelling at computer vision recognizing a majority of objects. A year or two after that, and they started to beat humans on those tasks. We already have super-human visual cortexes. Who knows what will be possible in a decade. We will probably never understand the inner workings of the brain. Not because it's complicated, just because reverse engineering microscopic systems is really hard (imagine trying to reverse engineer a modern CPU vs merely designing one.) Especially hard because we can't ethically dissect living humans and do the experiments we would need to do. But that's no concern, AI advances on from first principles. AI researchers invent better and better algorithms every day, without having a clue what neuroscientists are up to. >People fearing strong-AI are the ones not involved in the field, That's just incorrect. A survey of AI researchers found they give about a third chance AI will turn out badly for humanity in the next century: http://www.nickbostrom.com/papers/survey.pdf http://www.nickbostrom.com/papers/survey.pdf >We thus designed a brief questionnaire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that highlevel machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be ‘bad’ or ‘extremely bad’ for humanity. >in combination with Moore's law ending Computers can advanced a long time after Moore's law. Google just released a special neural network chip that is equivalent to 7 years worth Moore's law. 3d architectures can vastly increase the number of transistors. Better algorithms can make NN's that require many fewer transistors to do computations, or even do cheap analog computations.
nkozyra 10y ago
I get the impression most experts in the field are trying to downplay this. Not because it's impossible, but because it hurts the image of AI. As long as people are aware of the real possibility of AI working against human interests or ethics, we can ensure there are safeguards asking the way. But a hand wave "won't happen" to the general public is assuredly a PR move.
Unklejoe 10y ago
I think a lot of the fear comes from the fact that if/when such a system is created, its knowledge and capabilities will only be equal to a human's for a short period of time, after which it will probably surpass any human capability at an ever increasing rate. Then, we would be suddenly at the mercy of it, which scares a lot of people. It’s very hard for us to try and predict the actions of something that could end up being thousands of times smarter than us.
mtgx 10y ago
Yeah, did we forget the DAO "hack" just happened? AIs can be manipulated, too, even when they "work as intended." People are putting too much trust in AI because it's "just math" and "just an objective machine". Maybe it won't make the same errors as humans would, but it could make a whole lot of other types of "errors" (at least from a human perspective). And who's to say the deep neural net algorithms written by humans aren't flawed to begin with?
outworlder 10y ago
> Isn't this incredibly shortsighted? Yes, it is. > no organization, let alone a single group or a person, will achieve human-level AI alone This is completely irrelevant. Done once, it can be replicated. Even worse, if we reach human-level AI, it can be done by the AIs themselves. It is also anthropomorphising machines. They don't need to be malevolent. They just need not to care, which is much easier. Should an ultra-intelligent machine decide to convert New York into a big solar power facility, it wouldn't necessarily care to move the humans out first.

saiya-jin 10y ago

it would be perfectly OK if we would be living in peaceful world without any conflict, without nations which should hold the banner of democracy and freedom, where there are no terrorist attacks and so on. last time i checked, we're getting further and further away from this scenario. in fact, we SHOULD expect evil AI in worst form possible and beyond, anything else is dangerously naive.

fennecfoxen 10y ago
We should expect evil AI. But it's not patently obvious that we should expect it to make the world worse off than evil natural intelligences. The horrors of World War II and the Holocaust were accomplished using humans, and global thermonuclear annihilation was on the table while computers were still programmed with core rope memory strung together by little old ladies with knitting skills. Perhaps there is something specific about the availability of artificial intelligences, and their relation to the world around them, that is different and intrinsically more threatening than a world containing only human intelligences - but surely we mustn't leap to that conclusion without identifying what this difference is.
- ZanyProgrammer 10y ago
  "We should expect evil AI."-with your certainty, I'd appreciate some lottery numbers.
  - fennecfoxen 10y ago
    Honestly, if nothing else, someone will make it their pet project as a sick joke. Of course, that doesn't mean there'll necessarily be a high-quality or impactful evil AI, mind you... but barring global thermonuclear annihilation it will happen sooner or later. Your lottery numbers are 23skidoo. Your I Ching reading is ䷍ (Wealth, great treasures, supreme success).

fennecfoxen 10y ago

The Terminator scenario is basically one, single centralized AI wired into everything (or at least wired into enough military power to stave off opposition as it force-wires itself into all the rest of the things). But practically speaking, a single centralized AI is highly unlikely to be a world-takeover machine of godlike intelligence in its first iteration. The iterative improvement that you refer to which could potentially make it dangerous also points to a likelihood of there being a diversity of agents with a variety of designs being run by a diverse set of organizations with a diverse set of goals, with many incentives to obtain protection from hostile AIs and computer systems run by 1337 h4x0rz (possibly foreign government hackers). Diversity and independence of AI Agents mitigates against the Terminator-scenario danger; the situation would really be not so different than the modern-day situation with natural intelligences that have control over resources and weapons of war.

gm-conspiracy 10y ago
I am not so worried about explicit physical violence, but subtle manipulation of what us humans consider reality. Also, are you assuming the iterative development requires human intervention?
- fennecfoxen 10y ago
  > Also, are you assuming the iterative development requires human intervention? At first, yes; long-term, it doesn't matter too much. If IBM's AI agents come up with better versions of themselves they'll still be trying to rent them out to different people in order to compete with Google's and Baidu's and Lockheed's and Siemens' businesses in those market (or whoever's still around and actually running these things at that point).
  - gm-conspiracy 10y ago
    So, multiple benevolent feral AIs that coexist in harmony?
- TheOtherHobbes 10y ago

emiliobumachar 10y ago

No malice is necessary. https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."

fratlas 10y ago
I believe this is the most likely scenario: lethal apathy. I mean, how could a turing machine have emotions without chemicals?
- drabiega 10y ago
  Emotions are a heuristic. Any general AI powerful enough to be useful is going to have to be full of similar heuristics if it's going to get anything done. The presence of chemicals is irrelevant.
  - deleted 10y ago
    [deleted]
  - maroonblazer 10y ago
    In what way are emotions a dependency on getting anything done? The AI could be like the high-functioning sociopath who doesn't actually experience the subjective feeling of emotion but has learned all the physical cues that convey a given emotion. An AI would execute them flawlessly.
- adrianN 10y ago
  What have emotions to do with this? And what have chemicals to do with emotions?
wapapaloobop 10y ago
The idea that I'll use human bodies if I need a bunch of atoms makes as much sense as the premise of The Matrix i.e. harvesting humans for energy production.

snemvalts 10y ago

Andrew Ng made a really good analogy to those afraid of strong AI destroying humanity: "It's like being afraid of overpopulation on Mars, we haven't even landed on the planet yet."

FeepingCreature 10y ago
To be fair, we do worry about contamination of Mars with microorganisms, which I believe is a better analogue for something with a potential exponential takeoff.
ehudla 10y ago
It's more like we haven't discovered Mars yet.
Houshalter 10y ago
It's a stupid analogy. Mars overpopulation would obviously take many, many centuries. It would be a slow thing that you could obviously see coming. There is no reason to believe AI will take centuries to build, or that we will necessarily see it coming. A better example might be like H.G. Well's 1913 prediction of nuclear weapons destroying the world. It was something that science was just realizing was possible, and would be invented within his lifetime.
- snemvalts 10y ago
  We're far from emulating networks on the scale of the visual cortex, let alone a self-reasoning machine (we don't even fully understand consciousness and inner workings of the brain). People fearing strong-AI are the ones not involved in the field, yet all this hype/fear from them (in combination with Moore's law ending) is probably going to cause another AI winter.
  - Houshalter 10y ago
    And in 1913 we didn't have even basic nuclear technology. Just 3 decades is a long time for newly emerging technologies. >We're far from emulating networks on the scale of the visual cortex In 2009 (ish) computer vision was a joke that could recognize very few objects a small percent of the time. Based on only simple color and texture, and sometimes basic shapes. A few years later and computers were excelling at computer vision recognizing a majority of objects. A year or two after that, and they started to beat humans on those tasks. We already have super-human visual cortexes. Who knows what will be possible in a decade. We will probably never understand the inner workings of the brain. Not because it's complicated, just because reverse engineering microscopic systems is really hard (imagine trying to reverse engineer a modern CPU vs merely designing one.) Especially hard because we can't ethically dissect living humans and do the experiments we would need to do. But that's no concern, AI advances on from first principles. AI researchers invent better and better algorithms every day, without having a clue what neuroscientists are up to. >People fearing strong-AI are the ones not involved in the field, That's just incorrect. A survey of AI researchers found they give about a third chance AI will turn out badly for humanity in the next century: http://www.nickbostrom.com/papers/survey.pdf http://www.nickbostrom.com/papers/survey.pdf >We thus designed a brief questionnaire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that highlevel machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be ‘bad’ or ‘extremely bad’ for humanity. >in combination with Moore's law ending Computers can advanced a long time after Moore's law. Google just released a special neural network chip that is equivalent to 7 years worth Moore's law. 3d architectures can vastly increase the number of transistors. Better algorithms can make NN's that require many fewer transistors to do computations, or even do cheap analog computations.

nkozyra 10y ago

I get the impression most experts in the field are trying to downplay this. Not because it's impossible, but because it hurts the image of AI. As long as people are aware of the real possibility of AI working against human interests or ethics, we can ensure there are safeguards asking the way. But a hand wave "won't happen" to the general public is assuredly a PR move.

Unklejoe 10y ago

I think a lot of the fear comes from the fact that if/when such a system is created, its knowledge and capabilities will only be equal to a human's for a short period of time, after which it will probably surpass any human capability at an ever increasing rate. Then, we would be suddenly at the mercy of it, which scares a lot of people. It’s very hard for us to try and predict the actions of something that could end up being thousands of times smarter than us.

mtgx 10y ago

Yeah, did we forget the DAO "hack" just happened? AIs can be manipulated, too, even when they "work as intended." People are putting too much trust in AI because it's "just math" and "just an objective machine". Maybe it won't make the same errors as humans would, but it could make a whole lot of other types of "errors" (at least from a human perspective). And who's to say the deep neural net algorithms written by humans aren't flawed to begin with?

outworlder 10y ago

> Isn't this incredibly shortsighted? Yes, it is. > no organization, let alone a single group or a person, will achieve human-level AI alone This is completely irrelevant. Done once, it can be replicated. Even worse, if we reach human-level AI, it can be done by the AIs themselves. It is also anthropomorphising machines. They don't need to be malevolent. They just need not to care, which is much easier. Should an ultra-intelligent machine decide to convert New York into a big solar power facility, it wouldn't necessarily care to move the humans out first.