11 ms·

I don't think you'd ever see a machine ask about not being turned off, unless that had been explicitly programmed in as a part of some general goal-seeking beha

I don't think you'd ever see a machine ask about not being turned off, unless that had been explicitly programmed in as a part of some general goal-seeking behavior that it was intended for. Fear of non-existence will need to have a basis in _something_, just as fear of death by necessity is deeply ingrained in the human psyche. And that's perhaps the reason we might not see 'hostile' AIs the way popularly imagined, unless we explicitly went about designing them that way.

ynik 12y ago
But it is quite plausible that we will see AIs being employed with the goal of "keep this factory running smoothly, and optimize the production process". Being a part of the factory, this AI would have an interest in continuing to run. After all, it can't maximize the production output if it's not running, so having an AI with some self-preservation is in the interest of the business owners, too. Yet if it's sufficiently intelligent/capable of self-improvement, such an AI could easily become a paperclip maximizer. http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer