4 ms·

> [combination of things necessary for an AI to become a threat to humanity] Sound reasonable to you? Me either The question isn't whether it "sounds reasonabl

by azakai 12y ago

> [combination of things necessary for an AI to become a threat to humanity] Sound reasonable to you? Me either The question isn't whether it "sounds reasonable". If there is even a 2% chance of that happening, we need to consider it. Anything that can end humanity is worth considering even if it is unlikely. > Even if this were the case, there is absolutely no reason to believe that, by virtue of running on a computer, an AI will be better at computers than we are. No, there is every reason to believe that. Once we make an AI that is equal to us, we will be able to scale it up by running it on the next generation of CPUs, or running more such CPUs in parallel. We can't scale up human intelligence in any similar way. If an AI can match us, it can far exceed us. edit: in fact, the AI may well work to scale itself up

ikeboy 12y ago
I think the list is misleading and is not necessary for an AI to be dangerous. >It has an “I,” a sense of self distinct from others. I deny this is required for a program to be dangerous. >It has the intellectual capacity to step outside of the boundaries of its intended purpose and programming to form radically new goals for itself (the “I”). Deny this as well; even carrying out programmed goals can be harmful. In the words of Superintelligence (copied from https://intelligence.org/2015/01/08/brooks-searle-agi-volition-timelines/ https://intelligence.org/2015/01/08/brooks-searle-agi-voliti...): >[W]e cannot blithely assume that a superintelligence with the final goal of calculating the decimals of pi (or making paperclips, or counting grains of sand) would limit its activities in such a way as not to infringe on human interests. An agent with such a final goal would have a convergent instrumental reason, in many situations, to acquire an unlimited amount of physical resources and, if possible, to eliminate potential threats to itself and its goal system. >It has access to resources on a global scale to carry out the plan. This sounds reasonable. As http://slatestarcodex.com/2015/04/07/no-physical-substrate-no-problem/ http://slatestarcodex.com/2015/04/07/no-physical-substrate-n... points out, humans have made billions anonymously, just by being connected to the internet. http://lesswrong.com/lw/k4h/request_for_concrete_ai_takeover_mechanisms/ http://lesswrong.com/lw/k4h/request_for_concrete_ai_takeover... gives some ways an AI could takeover.
pgodzin 12y ago
Sure we should consider it, and the people working on it are considering it. But that doesn't mean we are at a point where we should do something drastic like regulation that would slow down research (Sam Altman's idea).

by azakai 12y ago

ikeboy 12y ago

I think the list is misleading and is not necessary for an AI to be dangerous. >It has an “I,” a sense of self distinct from others. I deny this is required for a program to be dangerous. >It has the intellectual capacity to step outside of the boundaries of its intended purpose and programming to form radically new goals for itself (the “I”). Deny this as well; even carrying out programmed goals can be harmful. In the words of Superintelligence (copied from https://intelligence.org/2015/01/08/brooks-searle-agi-volition-timelines/ https://intelligence.org/2015/01/08/brooks-searle-agi-voliti...): >[W]e cannot blithely assume that a superintelligence with the final goal of calculating the decimals of pi (or making paperclips, or counting grains of sand) would limit its activities in such a way as not to infringe on human interests. An agent with such a final goal would have a convergent instrumental reason, in many situations, to acquire an unlimited amount of physical resources and, if possible, to eliminate potential threats to itself and its goal system. >It has access to resources on a global scale to carry out the plan. This sounds reasonable. As http://slatestarcodex.com/2015/04/07/no-physical-substrate-no-problem/ http://slatestarcodex.com/2015/04/07/no-physical-substrate-n... points out, humans have made billions anonymously, just by being connected to the internet. http://lesswrong.com/lw/k4h/request_for_concrete_ai_takeover_mechanisms/ http://lesswrong.com/lw/k4h/request_for_concrete_ai_takeover... gives some ways an AI could takeover.

pgodzin 12y ago

Sure we should consider it, and the people working on it are considering it. But that doesn't mean we are at a point where we should do something drastic like regulation that would slow down research (Sam Altman's idea).