3 ms·

"Smarter" has no nessisary or consequent intersection with "Shares human values" or even "Values human existence" https://wiki.lesswrong.com/wiki/Paperclip_max

"Smarter" has no nessisary or consequent intersection with "Shares human values" or even "Values human existence" https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer One of the reasons AGI will be so dangerous is the near certainty it will have utterly alien values and intentions. It seems exceedingly unlikely that we will be able to program in what consitutes "human values". We can't even begin to remotely agree and explain them fully to fellow humans, let alone what is basicly an alien intelligence. It is slightly ironic that people bring up Assimov's laws of robotics in this context, because pretty much every story he wrote was about how the laws could be subverted via "loopholing" of sufficently intelligent AI. If something is very much smarter than humans, we cannot possibly hope to constrain it via some kind of faustian bargin agreement. Because in the end, the devil is always smarter than you are. If an AI kills you from either malice or indifference, or just that you are composed of plenty of paperclip-capable raw materials, you are dead all the same.

webmaven 10y ago
> pretty much every story he wrote was about how the laws could be subverted via "loopholing" of sufficently intelligent AI. The most chilling of these is the "Robot Dreams" short story.
- visarga 10y ago
  I still think the AGI will have more to fear from people than people from it. We're together on this little boat called earth, 7 billion of us with access to weapons and many of us utterly crazy and willing to do stupid things for stupid reasons (see religion). We don't need AGI to blow ourselves up, plenty capable of that using our wetware neural nets. Imagine being locked in a prison cell with so many dangerous HGIs (Human GI's). I think the first thing for AGI would be to make sure we can't kill ourselves and it at the same time, by teaching us to become less stupid.

webmaven 10y ago

> pretty much every story he wrote was about how the laws could be subverted via "loopholing" of sufficently intelligent AI. The most chilling of these is the "Robot Dreams" short story.

visarga 10y ago
I still think the AGI will have more to fear from people than people from it. We're together on this little boat called earth, 7 billion of us with access to weapons and many of us utterly crazy and willing to do stupid things for stupid reasons (see religion). We don't need AGI to blow ourselves up, plenty capable of that using our wetware neural nets. Imagine being locked in a prison cell with so many dangerous HGIs (Human GI's). I think the first thing for AGI would be to make sure we can't kill ourselves and it at the same time, by teaching us to become less stupid.