It's not so much that the super-intelligence can't question the usefulness of paperclips. It probably could. It could recognize that these paperclips aren't pro

It's not so much that the super-intelligence can't question the usefulness of paperclips. It probably could. It could recognize that these paperclips aren't providing anyone else value. It may recognize that its human creators actually don't want all these paperclips. It may be able to fully understand and internalize the humans saying "stop turning our atoms into paperclips!" But it doesn't care. What it cares about is maximizing paperclips. From some universal standpoint, there isn't something inherently great about humans, except that we're humans and our utility function usually means that we would prefer to go on living. By what standard would it be better to be survived by machines than by ourselves? Probably nothing more universal than the belief "humanity should persist in some recognizable state." I'd recommend reading "The genie knows, but doesn't care" (http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/ http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/): >If an artificial intelligence is smart enough to be dangerous, we'd intuitively expect it to be smart enough to know how to make itself safe. But that doesn't mean all smart AIs are safe. To turn that capacity into actual safety, we have to program the AI at the outset — before it becomes too fast, powerful, or complicated to reliably control — to already care about making its future self care about safety. That means we have to understand how to code safety. We can't pass the entire buck to the AI, when only an AI we've already safety-proofed will be safe to ask for help on safety issues! Given the five theses, this is an urgent problem if we're likely to figure out how to make a decent artificial programmer before we figure out how to make an excellent artificial ethicist.