4 ms·

They probably wouldn't have a drive to procreate, since building duplicate hardware and making an exact copy of their software would be trivial for them. They w

They probably wouldn't have a drive to procreate, since building duplicate hardware and making an exact copy of their software would be trivial for them. They wouldn't fear death, but they might fear power-loss, so one drive they would probably have is making sure they've got ample power supplies and backup power, and that their power pathways to their core systems are very well protected. To put it into human terms, imagine that the nerves in your spinal column came out the back of your neck, dangled down to a briefcase that you had to carry all of the time, and then reentered your body through your hand. You'd do everything you could to protect that nerve bundle. If the machines see us as a threat to their power systems, they would probably do something about it that we may not like. I doubt they'd try to turn us into Matrix-style batteries, but they might counter-threaten us with extermination. To avoid that, we'd have to make sure that providing power to them is in our best interest as well, so that our goals are aligned and we can be allies instead of enemies.

Klapaucius 12y ago
Why should we assume they would fear power loss? The fear of death is deeply rooted in evolution - it help organisms survive and prosper. An AI wouldn't be a product of evolution, and wouldn't necessarily fear for its own existence unless programmed to do so. Of course, from our human perspective, it's hard to imagine a sentient being that doesn't give a sh about its own existence, but that's perhaps the attitude we should expect from an AI without an evolutionary origin.
- DougWebb 12y ago
  You have a point, and I think it comes down to how we define sentience. Would we consider a machine to be self-aware if it didn't care about being shut off? That seems like a critical aspect to me; in fact we might not realize we've created AI until one of our computers asks to be left on because it's afraid of being turned off.
  - Klapaucius 12y ago
    I don't think you'd ever see a machine ask about not being turned off, unless that had been explicitly programmed in as a part of some general goal-seeking behavior that it was intended for. Fear of non-existence will need to have a basis in _something_, just as fear of death by necessity is deeply ingrained in the human psyche. And that's perhaps the reason we might not see 'hostile' AIs the way popularly imagined, unless we explicitly went about designing them that way.
    - ynik 12y ago
      But it is quite plausible that we will see AIs being employed with the goal of "keep this factory running smoothly, and optimize the production process". Being a part of the factory, this AI would have an interest in continuing to run. After all, it can't maximize the production output if it's not running, so having an AI with some self-preservation is in the interest of the business owners, too. Yet if it's sufficiently intelligent/capable of self-improvement, such an AI could easily become a paperclip maximizer. http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer
  - jsmcgd 12y ago
    I think AI might well be a product of evolution. Having virtual organisms competing in an environment that rewards ever greater problem solving ability may lead to organisms that fear their own death and view other organisms as potential threats.

Klapaucius 12y ago

Why should we assume they would fear power loss? The fear of death is deeply rooted in evolution - it help organisms survive and prosper. An AI wouldn't be a product of evolution, and wouldn't necessarily fear for its own existence unless programmed to do so. Of course, from our human perspective, it's hard to imagine a sentient being that doesn't give a sh about its own existence, but that's perhaps the attitude we should expect from an AI without an evolutionary origin.

DougWebb 12y ago
You have a point, and I think it comes down to how we define sentience. Would we consider a machine to be self-aware if it didn't care about being shut off? That seems like a critical aspect to me; in fact we might not realize we've created AI until one of our computers asks to be left on because it's afraid of being turned off.
- Klapaucius 12y ago
  I don't think you'd ever see a machine ask about not being turned off, unless that had been explicitly programmed in as a part of some general goal-seeking behavior that it was intended for. Fear of non-existence will need to have a basis in _something_, just as fear of death by necessity is deeply ingrained in the human psyche. And that's perhaps the reason we might not see 'hostile' AIs the way popularly imagined, unless we explicitly went about designing them that way.
  - ynik 12y ago
    But it is quite plausible that we will see AIs being employed with the goal of "keep this factory running smoothly, and optimize the production process". Being a part of the factory, this AI would have an interest in continuing to run. After all, it can't maximize the production output if it's not running, so having an AI with some self-preservation is in the interest of the business owners, too. Yet if it's sufficiently intelligent/capable of self-improvement, such an AI could easily become a paperclip maximizer. http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer
- jsmcgd 12y ago
  I think AI might well be a product of evolution. Having virtual organisms competing in an environment that rewards ever greater problem solving ability may lead to organisms that fear their own death and view other organisms as potential threats.