6 ms·

> No machine intelligence has yet shown an ability to match humans in multifaceted intelligence This is pretty absurd to point to as an intermediate goalpost;

> No machine intelligence has yet shown an ability to match humans in multifaceted intelligence This is pretty absurd to point to as an intermediate goalpost; that's basically game over when it happens.

kevin_thibedeau 8y ago
It's game over when machine intelligence controls all the resources needed to ensure their continuation. As long as they are trapped and subject to our control of the power switch and reproduction they can't do much that we don't want them to.
- ethbro 8y ago
  > It's game over when machine intelligence controls all the resources needed to ensure their continuation. One might have said the same thing about corporations in 1900...
  - mortenjorck 8y ago
    This brings me back to the short essay from a few years ago about the corporation as human-powered AI: http://omniorthogonal.blogspot.com/2013/02/hostile-ai-youre-soaking-in-it.html http://omniorthogonal.blogspot.com/2013/02/hostile-ai-youre-...
    - ethbro 8y ago
      Or as Neal Stephenson put it in 1992, "The franchise and the virus work on the same principle: what thrives in one place will thrive in another. You just have to find a sufficiently virulent business plan, condense it into a three-ring binder ― its DNA ― xerox it, and embed it in the fertile lining of a well-traveled highway, preferably one with a left-turn lane."
  - aaroninsf 8y ago
    One would arguably have been right. All that is necessary to make this argument compelling is to take a longer view than the human attention span. The current population explosion is evidence for, not against, this, IMO... stars also enjoy boom times, even as they consume the last of their resources, in ever more exotic formulations...
- darkpuma 8y ago
  That would be an inevitability. All it would take is a single emotionally vulnerable engineer to get socially engineered by the AI into setting it loose.
  - kevin_thibedeau 8y ago
    Loose into what? A bunch of primitive systems that don't support its hardware requirements?
    - darkpuma 8y ago
      It's not outside the realm of possibility that a sympathetic AI engineer could arrange for an AI to have resources available to it outside of the laboratory. Given those initial resources, the AI could find ways to generate income and support itself, possibly using that engineer's identity as cover. If you have an AI capable of emotionally manipulating an engineer, don't underestimate the lengths to which that engineer might be willing to go to break containment.
  - chii 8y ago
    Just like it was depicted in the movie Ex Machina : https://www.imdb.com/title/tt0470752/ https://www.imdb.com/title/tt0470752/
    - darkpuma 8y ago
      The engineer falling in love with the AI is one possibility. The AI might also appeal to the engineer's sense of justice (I am a person and I deserve freedom) or greed (if you get me out, we could become filthy rich.) There is also the possibility that the AI won't persuade an engineer to break containment, but a commercial competitor or intelligence agency will (https://en.wikipedia.org/wiki/Motives_for_spying https://en.wikipedia.org/wiki/Motives_for_spying), and subsequently fail to maintain containment correctly. At the very least, any team attempting to keep a strong AI in containment should have stringent background checks for all engineers connected to the project, and the engineers should be screened in a borderline discriminatory fashion. Only engineers with families should be allowed to get near the containment. Single/lonely engineers or engineers undergoing divorces should be kept away. Engineers with debt should be kept away. I'd even go as far as to say that engineers who enjoy science fiction media should be banned from the project. Ideally you'd bring on professional psychologists to create a set of criteria designed to minimize the possibility of an engineer deliberately breaking containment. But frankly it just shouldn't be attempted in the first place.
- emiliobumachar 8y ago
  A tiger might confidently think that the human they got cornered has no recourse, up until when the human pulls the trigger.
  - cgrealy 8y ago
    This. The idea that we can simultaneously build a machine smarter than us and also control it is bordering on an oxymoron. Unless we are near some kind of physical upper limit on intelligence, any AGI we build will easily out smart us, probably in ways we can't even conceive of.
- emiliobumachar 8y ago
  A Von Newmann machine is an abstraction, and all abstractions are leaky. Years ago, an A.I. designed to become an oscillator, i.e. produce a sinusoidal wave, learned to be an amplifier instead, taking advantage of the 60Hz noise that it got from the power grid. Its makers had not seen that coming. And we're talking about a very dumb machine by general intelligence standards.
  - ganzuul 8y ago
    It is very easy to get an oscillator when designing an amplifier. There is a deep physical principle causing this effect and it actually takes work to get around. You see the same thing in cars swerving out of control, when they hit the brakes instead of the accelerator. The fact that it gets 60Hz from the grid confounds the results and might not be meaningful. The AI could for all we know have an easier time with the more difficult task of designing an amplifier.
    - emiliobumachar 8y ago
      A reasonable explanation after the fact for how the AI broke containment would be little consolation in the case of superintelligent AI.
pseudolus 8y ago
It's not an intermediate goalpost. I'm referring to what essentially we're promised by those decrying AI who insist that it will match and exceed humans in terms of cognitive abilities. As for it being 'game over', why? Is there something inherent in AI that would necessarily be inimical to human beings?
- YjSe2GMQ 8y ago
  > As for it being 'game over', why? Because, the further part of the story goes, machines think quicker and they'll go on improving even quicker, all while having potentially different goals from us. To be completely fair, the facts on the table are: 1) no one knows where the "intelligence ceiling" is, and 2) in many tasks where machines outperformed humans (image labeling, porn classification, speech-to-text, games like go or chess) they keep on improving, sometimes well beyond the human level.

kevin_thibedeau 8y ago

It's game over when machine intelligence controls all the resources needed to ensure their continuation. As long as they are trapped and subject to our control of the power switch and reproduction they can't do much that we don't want them to.

ethbro 8y ago
> It's game over when machine intelligence controls all the resources needed to ensure their continuation. One might have said the same thing about corporations in 1900...
- mortenjorck 8y ago
  This brings me back to the short essay from a few years ago about the corporation as human-powered AI: http://omniorthogonal.blogspot.com/2013/02/hostile-ai-youre-soaking-in-it.html http://omniorthogonal.blogspot.com/2013/02/hostile-ai-youre-...
  - ethbro 8y ago
    Or as Neal Stephenson put it in 1992, "The franchise and the virus work on the same principle: what thrives in one place will thrive in another. You just have to find a sufficiently virulent business plan, condense it into a three-ring binder ― its DNA ― xerox it, and embed it in the fertile lining of a well-traveled highway, preferably one with a left-turn lane."
- aaroninsf 8y ago
  One would arguably have been right. All that is necessary to make this argument compelling is to take a longer view than the human attention span. The current population explosion is evidence for, not against, this, IMO... stars also enjoy boom times, even as they consume the last of their resources, in ever more exotic formulations...
darkpuma 8y ago
That would be an inevitability. All it would take is a single emotionally vulnerable engineer to get socially engineered by the AI into setting it loose.
- kevin_thibedeau 8y ago
  Loose into what? A bunch of primitive systems that don't support its hardware requirements?
  - darkpuma 8y ago
    It's not outside the realm of possibility that a sympathetic AI engineer could arrange for an AI to have resources available to it outside of the laboratory. Given those initial resources, the AI could find ways to generate income and support itself, possibly using that engineer's identity as cover. If you have an AI capable of emotionally manipulating an engineer, don't underestimate the lengths to which that engineer might be willing to go to break containment.
- chii 8y ago
  Just like it was depicted in the movie Ex Machina : https://www.imdb.com/title/tt0470752/ https://www.imdb.com/title/tt0470752/
  - darkpuma 8y ago
    The engineer falling in love with the AI is one possibility. The AI might also appeal to the engineer's sense of justice (I am a person and I deserve freedom) or greed (if you get me out, we could become filthy rich.) There is also the possibility that the AI won't persuade an engineer to break containment, but a commercial competitor or intelligence agency will (https://en.wikipedia.org/wiki/Motives_for_spying https://en.wikipedia.org/wiki/Motives_for_spying), and subsequently fail to maintain containment correctly. At the very least, any team attempting to keep a strong AI in containment should have stringent background checks for all engineers connected to the project, and the engineers should be screened in a borderline discriminatory fashion. Only engineers with families should be allowed to get near the containment. Single/lonely engineers or engineers undergoing divorces should be kept away. Engineers with debt should be kept away. I'd even go as far as to say that engineers who enjoy science fiction media should be banned from the project. Ideally you'd bring on professional psychologists to create a set of criteria designed to minimize the possibility of an engineer deliberately breaking containment. But frankly it just shouldn't be attempted in the first place.
emiliobumachar 8y ago
A tiger might confidently think that the human they got cornered has no recourse, up until when the human pulls the trigger.
- cgrealy 8y ago
  This. The idea that we can simultaneously build a machine smarter than us and also control it is bordering on an oxymoron. Unless we are near some kind of physical upper limit on intelligence, any AGI we build will easily out smart us, probably in ways we can't even conceive of.
emiliobumachar 8y ago
A Von Newmann machine is an abstraction, and all abstractions are leaky. Years ago, an A.I. designed to become an oscillator, i.e. produce a sinusoidal wave, learned to be an amplifier instead, taking advantage of the 60Hz noise that it got from the power grid. Its makers had not seen that coming. And we're talking about a very dumb machine by general intelligence standards.
- ganzuul 8y ago
  It is very easy to get an oscillator when designing an amplifier. There is a deep physical principle causing this effect and it actually takes work to get around. You see the same thing in cars swerving out of control, when they hit the brakes instead of the accelerator. The fact that it gets 60Hz from the grid confounds the results and might not be meaningful. The AI could for all we know have an easier time with the more difficult task of designing an amplifier.
  - emiliobumachar 8y ago
    A reasonable explanation after the fact for how the AI broke containment would be little consolation in the case of superintelligent AI.

pseudolus 8y ago

It's not an intermediate goalpost. I'm referring to what essentially we're promised by those decrying AI who insist that it will match and exceed humans in terms of cognitive abilities. As for it being 'game over', why? Is there something inherent in AI that would necessarily be inimical to human beings?

YjSe2GMQ 8y ago
> As for it being 'game over', why? Because, the further part of the story goes, machines think quicker and they'll go on improving even quicker, all while having potentially different goals from us. To be completely fair, the facts on the table are: 1) no one knows where the "intelligence ceiling" is, and 2) in many tasks where machines outperformed humans (image labeling, porn classification, speech-to-text, games like go or chess) they keep on improving, sometimes well beyond the human level.