3 ms·

I've read many arguments concerned with machine intelligence and the risks it poses, but I haven't read that book. Does it tackle explaining why we should conc

I've read many arguments concerned with machine intelligence and the risks it poses, but I haven't read that book. Does it tackle explaining why we should conclude that a machine intelligence would even communicate with us, much less seize resources that we need? I rarely see it addressed, and have never seen it addressed to my own satisfaction; what it is about humans that would be worth the trouble of eliminating. I'm also really interested in the communication question. I do not see any clear reason why a machine intelligence would develop "individuals" or would develop the ability to communicate with us.

Houshalter 11y ago
Yes that's one of the important topics covered in the book. The AI probably would probably value resources and self preservation. Resources help it fulfil whatever it's goal is, and the more the better. That includes energy and computing power. Which it could gain a lot of by converting the mass of the solar system into a dyson swarm. Or covering the Earth with solar panels or fusion reactors and comuters. Etc. Self preservation would be optimized by destroying anything that has even a tiny chance of being a threat to it. Even if humans are harmless, we could build other AIs like it which might compete with it. It's best to just destroy us. The problem is the AI doesn't need to want to eliminate us. It will likely be entirely indifferent to us. We just need to be in it's way.
JoshTriplett 11y ago
> what it is about humans that would be worth the trouble of eliminating Among other reasons, we're made of matter that isn't actively contributing to its value function as efficiently as it could be. Any AI that doesn't actively value sapient beings will very likely destroy them as an unintentional side effect of whatever it intends to do.

3 ms·

I've read many arguments concerned with machine intelligence and the risks it poses, but I haven't read that book. Does it tackle explaining why we should conc

by otakucode 11y ago

Houshalter 11y ago
Yes that's one of the important topics covered in the book. The AI probably would probably value resources and self preservation. Resources help it fulfil whatever it's goal is, and the more the better. That includes energy and computing power. Which it could gain a lot of by converting the mass of the solar system into a dyson swarm. Or covering the Earth with solar panels or fusion reactors and comuters. Etc. Self preservation would be optimized by destroying anything that has even a tiny chance of being a threat to it. Even if humans are harmless, we could build other AIs like it which might compete with it. It's best to just destroy us. The problem is the AI doesn't need to want to eliminate us. It will likely be entirely indifferent to us. We just need to be in it's way.
JoshTriplett 11y ago
> what it is about humans that would be worth the trouble of eliminating Among other reasons, we're made of matter that isn't actively contributing to its value function as efficiently as it could be. Any AI that doesn't actively value sapient beings will very likely destroy them as an unintentional side effect of whatever it intends to do.