3 ms·
> Any thought like ‘what’s so great about paperclips anyway?’ would be judged as not likely to lead to more paperclips and so remain unexplored. As is alluded
by bambax 12y ago
> Any thought like ‘what’s so great about paperclips anyway?’ would be judged as not likely to lead to more paperclips and so remain unexplored.
As is alluded to in the article, but not really further explored, we humans reason exactly like this hypothetical paper-clip-production-maximizer machine. Our goal is to maximize the number of humans, whatever the consequences to anything non-human, living or dead -- and, as is becoming more and more obvious, regardless of the consequences to ourselves.
What's so great about humans anyway?
From a human point of view, humans are better than, say, tigers. But how would you convince a tiger that it's moralistically, absolutely "better" that they all die off so that humans can build concrete towers where they can watch TV on top of one another?
There's also a lot of hubris in saying a super-intelligence capable of turning the whole solar system into a giant paperclip factory, would be unable to question the usefulness of paperclips.
In the end we are all going to die anyway; isn't it better to be survived by some kind of all-powerful, all-knowing machine, than another long line of mediocre mean chimps like ourselves?
- DanAndersen 12y agoIt's not so much that the super-intelligence can't question the usefulness of paperclips. It probably could. It could recognize that these paperclips aren't providing anyone else value. It may recognize that its human creators actually don't want all these paperclips. It may be able to fully understand and internalize the humans saying "stop turning our atoms into paperclips!" But it doesn't care. What it cares about is maximizing paperclips. From some universal standpoint, there isn't something inherently great about humans, except that we're humans and our utility function usually means that we would prefer to go on living. By what standard would it be better to be survived by machines than by ourselves? Probably nothing more universal than the belief "humanity should persist in some recognizable state." I'd recommend reading "The genie knows, but doesn't care" (http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/ http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/): >If an artificial intelligence is smart enough to be dangerous, we'd intuitively expect it to be smart enough to know how to make itself safe. But that doesn't mean all smart AIs are safe. To turn that capacity into actual safety, we have to program the AI at the outset — before it becomes too fast, powerful, or complicated to reliably control — to already care about making its future self care about safety. That means we have to understand how to code safety. We can't pass the entire buck to the AI, when only an AI we've already safety-proofed will be safe to ask for help on safety issues! Given the five theses, this is an urgent problem if we're likely to figure out how to make a decent artificial programmer before we figure out how to make an excellent artificial ethicist.
- bambax 12y agoIt's funny that we should now fear "AI" as if the cause of all our troubles wasn't, rather, HI (or is it NI for Natural?) AMDs? Terrorism? Global Warming? Who's responsible for these? Some kind of dumb paperclip-producing computer, or us -- you and me?
- bequanna 12y ago> Our goal is to maximize the number of humans, whatever the consequences to anything non-human, living or dead -- and, as is becoming more and more obvious, regardless of the consequences to ourselves. Is it? I'm 30 and (so far) childless. If we have children, my wife and I will have 1 or 2 kids. 'Maximizing the number of humans' is not a goal of ours. Most industrialized nations are reproducing at or below replacement levels and global population will probably level off by the end of this century.
- seigo 12y ago> Most industrialized nations are reproducing at or below replacement levels and global population will probably level off by the end of this century. And most third-world countries have higher birth rates than replacement levels. I'm not sure what kind of conclusion can be taken from a situation where there are increasingly more uneducated people with increasingly better technologies.. But it doesn't sound good.