3 ms·

> I've seen multiple top people in AGI space say that total annihilation threat from AGI exists. You've seen them say it, yes. Have you considered how plausibl

> I've seen multiple top people in AGI space say that total annihilation threat from AGI exists. You've seen them say it, yes. Have you considered how plausible it actually is, or are you just taking their word for it? You better have some damn solid evidence before you start talking about "inquisitions", and I'm only seeing wild speculation. Which, to be fair, is how most historical inquisitions started, for all the good those did.

tim333 4y ago
It's not hard to see the danger if you think about it a little. Probably before long we'll have AGI at human levels. Then inevitably it will get to better than human levels and be able to improve itself without needing humans to so. The question then is could it turn on humans and wipe us out? I'm optimistic the last one won't happen but it's hard to say the probability is zero.
- andrewflnr 4y ago
  I've heard all this. What I've not heard is a plausible, concrete mechanism for it to happen. Only wild speculation. Questions: define "better then human levels"? Better in what respects? What is the mechanism by which it improves itself, especially one that isn't wholly reliant on human cooperation? What is the actual motive for "turning on humans"? Current AIs don't seem to have motives at all; it's kind of a bio-evolved thing. How will a suitably motivated AI get access to enough physical power to "wipe us out", bearing in mind that would be a tall order even for a major nuclear power going rogue, to say nothing if having to work through manipulated servants (and don't forget that "human level intelligences", or greater if you don't collaborations, armed with dedicated neural circuitry for empathy, have been working on how to manipulate people for literal millennia with spotty results. All of these things have to go exactly wrong for AGI to be what kills us. Something else will almost certainly get us first.

tim333 4y ago

It's not hard to see the danger if you think about it a little. Probably before long we'll have AGI at human levels. Then inevitably it will get to better than human levels and be able to improve itself without needing humans to so. The question then is could it turn on humans and wipe us out? I'm optimistic the last one won't happen but it's hard to say the probability is zero.

andrewflnr 4y ago
I've heard all this. What I've not heard is a plausible, concrete mechanism for it to happen. Only wild speculation. Questions: define "better then human levels"? Better in what respects? What is the mechanism by which it improves itself, especially one that isn't wholly reliant on human cooperation? What is the actual motive for "turning on humans"? Current AIs don't seem to have motives at all; it's kind of a bio-evolved thing. How will a suitably motivated AI get access to enough physical power to "wipe us out", bearing in mind that would be a tall order even for a major nuclear power going rogue, to say nothing if having to work through manipulated servants (and don't forget that "human level intelligences", or greater if you don't collaborations, armed with dedicated neural circuitry for empathy, have been working on how to manipulate people for literal millennia with spotty results. All of these things have to go exactly wrong for AGI to be what kills us. Something else will almost certainly get us first.