6 ms·

The Doomsday Invention

72 points by pc 11y ago

bhouston 11y ago
I instantly noticed a Three.JS / WebGL animated background. Sort of neat. I think it is by Jono Brandel: http://about.jonobr1.com/ http://about.jonobr1.com/
sgt101 11y ago
Jaron Lanier is quoted here, but his compelling (to me) arguments on this topic are not aired. I recommend : http://edge.org/conversation/jaron_lanier-the-myth-of-ai http://edge.org/conversation/jaron_lanier-the-myth-of-ai As the gentleman says, a lack of Autonomous Agents is not the thing standing between humanity and oblivion. The conspicuous surplus of super empowered agents (as in "I could leave this room and 80 million people would be dead 40 minutes later") is. As for the Royal Society stuff. For the record it's not learning, and it's not the machines, I can say it again if anyone bothers to listen. Although given that it's not what will get news print or sell books or get on telly I am pretty sure that no one will. Ho hum. (edited 'cos I forgot a bracket, which was why I gave up c)
- rictic 11y ago
  That's a false dilemma. It would be wise to defend against every existential threat in proportion to its likelihood. The threat of nuclear war is well known, and significant resources are spent on reducing its likelihood. Until very recently the same could not be said for the threat of unfriendly AI. We can do both.
  - kansface 11y ago
    > That's a false dilemma. It would be wise to defend against every existential threat in proportion to its likelihood. Lives saved versus dollars spent is a better metric. Having said that, the Singularity crowd tends to lump in all potential future lives into the saved category. The question of allocating resources to avoid existential threats very quickly turns into a Pascal's mugging in favor of AI safety (think of the trillions of future children). > The threat of nuclear war is well known ... until very recently the same could not be said for the threat of unfriendly AI. Enough nuclear weapons exist to kill all humans forever. Strong AI does not exist. Expert estimates range from decades to over a century. At the same time, the people making the most noise seem come from the side of avoiding existential threats and not AI research. I am (highly) skeptical they will produce anything of value. Let me rephrase. Would you expect any productive work in 1840 by philosophers concerning potential dooms day weapons yet to be created by physicists?
    - vilhelm_s 11y ago
      There doesn't exist anywhere near enough nuclear weapons to kill all humans. Even in the 1980s, people were predicting between 20 and 100 million immediate American deaths from a full-scale Soviet attack, and since then there are much fewer nuclear weapons. See e.g. the discussion in this comment thread: https://slatestarcodex.com/2015/10/31/ot32-when-hell-is-full-the-thread-will-walk-the-earth/#comment-255524 https://slatestarcodex.com/2015/10/31/ot32-when-hell-is-full... It has been suggested that humans recovered from a population bottleneck of less than 30,000 individuals (the Toba catastrophy). A nuclear war would leave billions alive.
  - JoshTriplett 11y ago
    On the other hand, nuclear research could solve our energy needs for the foreseeable future. And AI could protect humanity against almost every possible existential risk. Let's hope that the increased notice taken of AI, and fear thereof, doesn't suddenly shut down a major source of hope for the future.
- ScottBurson 11y ago
  Very interesting read -- thanks for the link! Poking around a little, I also find this essay, which I think does a better job of getting to the root of the issue: http://edge.org/conversation/one-half-a-manifesto http://edge.org/conversation/one-half-a-manifesto
skybrian 11y ago
I'm wondering how much Eliezer Yudkowsky and the LessWrong crowd were influenced by Bostrom or vice versa. This all sounds very familiar, but the dates seem earlier: "Bostrom introduced the philosophical concept of 'existential risk' in 2002."
- eurg 11y ago
  Bostrom brought up the orthogonality idea before Eliezer, who consequently (not immediately) changed his mind. EDIT: I am unable to find the source. Might have been a halluzination, but I thought I remembered them having a discussion where Eliezer more or less still thought "let the AGI win, humanity be damned". Ah, my brain and I are not in a good relationship.
- Animats 11y ago
  "Bostrom introduced the philosophical concept of 'existential risk' in 2002." Oh, come on. This has been a theme in science fiction back to at least the 1940s. Jack Williamson's "With Folded Hands" (1947) is probably the clearest early writing on the subject. "The car turned off the shining avenue, taking him back to the quiet splendor of his prison. His futile hands clenched and relaxed again, folded on his knees. There was nothing left to do."
  - j1o1h1n 11y ago
    Also Frederic Brown, "Etaoin Shrdlu", 1942, which I have only read synopses of.
    - Animats 11y ago
      "It believes what it sets", about a sentient Linotype. That's a very funny short story. A magic story, though, not AI.
    - MichaelMoser123 11y ago
      Also Karel Čapek wrote R.U.R in 1920; (well its about assembled artificial biological beings, so you can say that it is not about AI). https://en.wikipedia.org/wiki/R.U.R https://en.wikipedia.org/wiki/R.U.R. And there is the Golem, which is even earlier. Also i find it funny how much attention this theoretical problem receives, when we don't have a good explanations on what intelligence is and how it works.
Animats 11y ago
Wasn't this covered on HN a few days ago?
- dang 11y ago
  It has been posted many times, but none of the posts rose above the 'significant attention' threshold described in the FAQ. This one does, though, so it'll be treated as a dupe if it gets posted again. https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html Btw, we're working on a better approach to duplicate detection that will reduce the number of reposts and will more often give credit to the original submitter.
flashman 11y ago
"Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb." A bomb is dumb, its explosion just raw chemistry. I would argue we're like small children playing with a sleeping parent: when it finally wakes up, it's going to have its own agenda for us.
- Filligree 11y ago
  That gets the 'intelligence', but misses the 'amoral'. Parents tend to have their children's best interests in mind. If we're playing this game, then humanity is more like a small child trying to wake up Cthulhu in hopes that he'll play nice.
  - Retra 11y ago
    You're going to build an amoral, powerful computer and it's going to do what? Slap the fork out of your hand every time you try to eat something? We build machines using a highly human-dependent artificial selection process. That's not going to change because the number one feature being selected for is usefulness to humans. I'm confused about how that's going to result in an even modestly-powerful, fully integrated machine that is not demonstrably useful to us. That's a bit like saying we should reconsider having sex because our babies might evolve grenades for hands, and that would be suicidal.
- TheOtherHobbes 11y ago
  Humans are dumb too, but still manage to explode regularly - although not usually with intelligence. I'm completely failing to understand why a hypothetical god-AI is being taken seriously as a threat to survival, but the many potential and actual horrors of the human world are being ignored here. We're perfectly capable of destroying ourselves without needing to invent an AI to do it for us. Why is this not taken seriously as a problem?
  - jimbokun 11y ago
    You should actually read the article. It directly discusses the questions you raise.
TazeTSchnitzel 11y ago
What makes artifical intelligence more likely to destroy humanity than a person?
- lifeformed 11y ago
  You can scale an AI's power more easily.
  - Chathamization 11y ago
    I'm not sure about that. A person can theoretically be scaled by adding another person to their cause, and there are a lot of people around. If the first brain in a box is simulated by a supercomputer that has only three others like it in existence, it's going to be much more difficult to scale it. Of course, just adding another person doesn't necessarily make things go faster (mythical man-month and all that). But then again, just adding more hardware isn't going to necessarily make a brain in a box more powerful, either.
    - lifeformed 11y ago
      I was thinking more in terms of cost. Adding a person to your cause requires a lot of money (the cost of them devoting their entire working time), and you only get so much extra benefit (efficiency losses due to communication and their own incentives). And the more effective the human, the more he would cost. With an AI, you just dump that money directly into the brain by slapping on more compute power and storage. It'd be a direct increase in ability to its will. I'm not sure what exactly the raw costs would be (maybe it would still cost more than hiring a human), but at least it seems like it'd be a much more efficient use of each dollar.
      - Chathamization 11y ago
        The supercomputer that just simulated one second's worth of brain activity (K computer) cost $1.25 billion to design and build and $10 million a year to operate. Now, I don't have experience with supercomputers, but I doubt that you'd just be able to slap in some extra ram and get a big speed up. It seems likely that the first simulated human level intelligences, at whatever point in the future they pop up, will at first be extremely costly and extremely slow. We don't really know if, say, connecting two supercomputers two simulate twice as many virtual neurons would make it more effective or if it would make it too unstable to do anything (how would a human brain with twice as many neurons react? It might just be mentally unstable and unproductive). But even a simple doubling experiment like that would probably take a long time to plan out and realize.
mirimir 11y ago
At least at first, the most powerful entities will be enhanced humans aka transhumans aka human-AI hybrids. Indeed, we're already there, in the sense that we rely on machines so heavily, and in so many ways. Interfaces will improve, with greater integration, of course. I've been reading Hannu Rajaniemi lately. He has in some ways a trippier take on this stuff than even Peter Watts does.
ecksor 11y ago
"the universe appears lifeless not because complex life is unusual but, rather, because it is always somehow thwarted before it becomes advanced enough to colonize space." The universe appears lifeless because it is a Dark Forest. http://www.goodreads.com/book/show/23168817-the-dark-forest http://www.goodreads.com/book/show/23168817-the-dark-forest
- ajmurmann 11y ago
  That was a great book. My only issue was the phoney science around locating where a signal comes from.
industriousthou 11y ago
I'm not really well read on this stuff, but do most people think that deep brain computer interfaces are farther off than self-improving artificial general intelligence? I usually see arguments about harmful AGI based on the idea that AGI will be able to optimize itself and acquire computational resources to become powerful in a scary way. I just sort of think that the human brain is already a tremendously powerful computational resource and augmenting it with weaker AI is going to be easier, cheaper, and ultimately, much powerful than AGI. But, of course that depends on the development of really powerful BCIs. Why not develop AI systems that complement the already insanely sophisticated computational system of the human brain instead of trying to replace it? Or enable groups of humans to work together along with AI to solve really hard problems? People often seem to think of AI as an arms race, but is it possible that systems that leverage human intelligence may have an insurmountable advantage?
- mirimir 11y ago
  I agree. Any system with potential for strong AI will be more immediately useful for augmentation. Although it's dated, I recommend Disappearing Through the Skylight O. B. Hardison, Jr.

72 points by pc 11y ago

I instantly noticed a Three.JS / WebGL animated background. Sort of neat. I think it is by Jono Brandel: http://about.jonobr1.com/ http://about.jonobr1.com/

sgt101 11y ago

Jaron Lanier is quoted here, but his compelling (to me) arguments on this topic are not aired. I recommend : http://edge.org/conversation/jaron_lanier-the-myth-of-ai http://edge.org/conversation/jaron_lanier-the-myth-of-ai As the gentleman says, a lack of Autonomous Agents is not the thing standing between humanity and oblivion. The conspicuous surplus of super empowered agents (as in "I could leave this room and 80 million people would be dead 40 minutes later") is. As for the Royal Society stuff. For the record it's not learning, and it's not the machines, I can say it again if anyone bothers to listen. Although given that it's not what will get news print or sell books or get on telly I am pretty sure that no one will. Ho hum. (edited 'cos I forgot a bracket, which was why I gave up c)

rictic 11y ago
That's a false dilemma. It would be wise to defend against every existential threat in proportion to its likelihood. The threat of nuclear war is well known, and significant resources are spent on reducing its likelihood. Until very recently the same could not be said for the threat of unfriendly AI. We can do both.
- kansface 11y ago
  > That's a false dilemma. It would be wise to defend against every existential threat in proportion to its likelihood. Lives saved versus dollars spent is a better metric. Having said that, the Singularity crowd tends to lump in all potential future lives into the saved category. The question of allocating resources to avoid existential threats very quickly turns into a Pascal's mugging in favor of AI safety (think of the trillions of future children). > The threat of nuclear war is well known ... until very recently the same could not be said for the threat of unfriendly AI. Enough nuclear weapons exist to kill all humans forever. Strong AI does not exist. Expert estimates range from decades to over a century. At the same time, the people making the most noise seem come from the side of avoiding existential threats and not AI research. I am (highly) skeptical they will produce anything of value. Let me rephrase. Would you expect any productive work in 1840 by philosophers concerning potential dooms day weapons yet to be created by physicists?
  - vilhelm_s 11y ago
    There doesn't exist anywhere near enough nuclear weapons to kill all humans. Even in the 1980s, people were predicting between 20 and 100 million immediate American deaths from a full-scale Soviet attack, and since then there are much fewer nuclear weapons. See e.g. the discussion in this comment thread: https://slatestarcodex.com/2015/10/31/ot32-when-hell-is-full-the-thread-will-walk-the-earth/#comment-255524 https://slatestarcodex.com/2015/10/31/ot32-when-hell-is-full... It has been suggested that humans recovered from a population bottleneck of less than 30,000 individuals (the Toba catastrophy). A nuclear war would leave billions alive.
- JoshTriplett 11y ago
  On the other hand, nuclear research could solve our energy needs for the foreseeable future. And AI could protect humanity against almost every possible existential risk. Let's hope that the increased notice taken of AI, and fear thereof, doesn't suddenly shut down a major source of hope for the future.
ScottBurson 11y ago
Very interesting read -- thanks for the link! Poking around a little, I also find this essay, which I think does a better job of getting to the root of the issue: http://edge.org/conversation/one-half-a-manifesto http://edge.org/conversation/one-half-a-manifesto

skybrian 11y ago

I'm wondering how much Eliezer Yudkowsky and the LessWrong crowd were influenced by Bostrom or vice versa. This all sounds very familiar, but the dates seem earlier: "Bostrom introduced the philosophical concept of 'existential risk' in 2002."

eurg 11y ago
Bostrom brought up the orthogonality idea before Eliezer, who consequently (not immediately) changed his mind. EDIT: I am unable to find the source. Might have been a halluzination, but I thought I remembered them having a discussion where Eliezer more or less still thought "let the AGI win, humanity be damned". Ah, my brain and I are not in a good relationship.
Animats 11y ago
"Bostrom introduced the philosophical concept of 'existential risk' in 2002." Oh, come on. This has been a theme in science fiction back to at least the 1940s. Jack Williamson's "With Folded Hands" (1947) is probably the clearest early writing on the subject. "The car turned off the shining avenue, taking him back to the quiet splendor of his prison. His futile hands clenched and relaxed again, folded on his knees. There was nothing left to do."
- j1o1h1n 11y ago
  Also Frederic Brown, "Etaoin Shrdlu", 1942, which I have only read synopses of.
  - Animats 11y ago
    "It believes what it sets", about a sentient Linotype. That's a very funny short story. A magic story, though, not AI.
  - MichaelMoser123 11y ago
    Also Karel Čapek wrote R.U.R in 1920; (well its about assembled artificial biological beings, so you can say that it is not about AI). https://en.wikipedia.org/wiki/R.U.R https://en.wikipedia.org/wiki/R.U.R. And there is the Golem, which is even earlier. Also i find it funny how much attention this theoretical problem receives, when we don't have a good explanations on what intelligence is and how it works.

Animats 11y ago

Wasn't this covered on HN a few days ago?

dang 11y ago
It has been posted many times, but none of the posts rose above the 'significant attention' threshold described in the FAQ. This one does, though, so it'll be treated as a dupe if it gets posted again. https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html Btw, we're working on a better approach to duplicate detection that will reduce the number of reposts and will more often give credit to the original submitter.

flashman 11y ago

"Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb." A bomb is dumb, its explosion just raw chemistry. I would argue we're like small children playing with a sleeping parent: when it finally wakes up, it's going to have its own agenda for us.

Filligree 11y ago
That gets the 'intelligence', but misses the 'amoral'. Parents tend to have their children's best interests in mind. If we're playing this game, then humanity is more like a small child trying to wake up Cthulhu in hopes that he'll play nice.
- Retra 11y ago
  You're going to build an amoral, powerful computer and it's going to do what? Slap the fork out of your hand every time you try to eat something? We build machines using a highly human-dependent artificial selection process. That's not going to change because the number one feature being selected for is usefulness to humans. I'm confused about how that's going to result in an even modestly-powerful, fully integrated machine that is not demonstrably useful to us. That's a bit like saying we should reconsider having sex because our babies might evolve grenades for hands, and that would be suicidal.
TheOtherHobbes 11y ago
Humans are dumb too, but still manage to explode regularly - although not usually with intelligence. I'm completely failing to understand why a hypothetical god-AI is being taken seriously as a threat to survival, but the many potential and actual horrors of the human world are being ignored here. We're perfectly capable of destroying ourselves without needing to invent an AI to do it for us. Why is this not taken seriously as a problem?
- jimbokun 11y ago
  You should actually read the article. It directly discusses the questions you raise.

TazeTSchnitzel 11y ago

What makes artifical intelligence more likely to destroy humanity than a person?

lifeformed 11y ago
You can scale an AI's power more easily.
- Chathamization 11y ago
  I'm not sure about that. A person can theoretically be scaled by adding another person to their cause, and there are a lot of people around. If the first brain in a box is simulated by a supercomputer that has only three others like it in existence, it's going to be much more difficult to scale it. Of course, just adding another person doesn't necessarily make things go faster (mythical man-month and all that). But then again, just adding more hardware isn't going to necessarily make a brain in a box more powerful, either.
  - lifeformed 11y ago
    I was thinking more in terms of cost. Adding a person to your cause requires a lot of money (the cost of them devoting their entire working time), and you only get so much extra benefit (efficiency losses due to communication and their own incentives). And the more effective the human, the more he would cost. With an AI, you just dump that money directly into the brain by slapping on more compute power and storage. It'd be a direct increase in ability to its will. I'm not sure what exactly the raw costs would be (maybe it would still cost more than hiring a human), but at least it seems like it'd be a much more efficient use of each dollar.
    - Chathamization 11y ago
      The supercomputer that just simulated one second's worth of brain activity (K computer) cost $1.25 billion to design and build and $10 million a year to operate. Now, I don't have experience with supercomputers, but I doubt that you'd just be able to slap in some extra ram and get a big speed up. It seems likely that the first simulated human level intelligences, at whatever point in the future they pop up, will at first be extremely costly and extremely slow. We don't really know if, say, connecting two supercomputers two simulate twice as many virtual neurons would make it more effective or if it would make it too unstable to do anything (how would a human brain with twice as many neurons react? It might just be mentally unstable and unproductive). But even a simple doubling experiment like that would probably take a long time to plan out and realize.

mirimir 11y ago

At least at first, the most powerful entities will be enhanced humans aka transhumans aka human-AI hybrids. Indeed, we're already there, in the sense that we rely on machines so heavily, and in so many ways. Interfaces will improve, with greater integration, of course. I've been reading Hannu Rajaniemi lately. He has in some ways a trippier take on this stuff than even Peter Watts does.

ecksor 11y ago

"the universe appears lifeless not because complex life is unusual but, rather, because it is always somehow thwarted before it becomes advanced enough to colonize space." The universe appears lifeless because it is a Dark Forest. http://www.goodreads.com/book/show/23168817-the-dark-forest http://www.goodreads.com/book/show/23168817-the-dark-forest

ajmurmann 11y ago
That was a great book. My only issue was the phoney science around locating where a signal comes from.

industriousthou 11y ago

I'm not really well read on this stuff, but do most people think that deep brain computer interfaces are farther off than self-improving artificial general intelligence? I usually see arguments about harmful AGI based on the idea that AGI will be able to optimize itself and acquire computational resources to become powerful in a scary way. I just sort of think that the human brain is already a tremendously powerful computational resource and augmenting it with weaker AI is going to be easier, cheaper, and ultimately, much powerful than AGI. But, of course that depends on the development of really powerful BCIs. Why not develop AI systems that complement the already insanely sophisticated computational system of the human brain instead of trying to replace it? Or enable groups of humans to work together along with AI to solve really hard problems? People often seem to think of AI as an arms race, but is it possible that systems that leverage human intelligence may have an insurmountable advantage?

mirimir 11y ago
I agree. Any system with potential for strong AI will be more immediately useful for augmentation. Although it's dated, I recommend Disappearing Through the Skylight O. B. Hardison, Jr.