4 ms·

If a good algorithm optimizes for "quantity of paperclips produced", then it would recognize that "prevent humans from turning me off right now" is an optimal s

• peterwwillis 9y ago
  The thought experiment is not realistic, because no intelligent system would think it could produce paperclips infinitely, and it would already have an "off feature" which was designed for a purpose (maintenance, update, replacement), and was intended to be used. An intelligent system would not reject proper use. We already replace humans at their jobs all the time and they only very rarely kill their masters to prevent it.
  • ric2b 9y ago
    > because no intelligent system would think it could produce paperclips infinitely This is irrelevant, being turned off while it's still possible to produce more paperclips would not be the optimal strategy for the machine trying to maximize paperclip production. > and it would already have an "off feature" which was designed for a purpose Look up what the control problem is. We don't know how to design such a feature for a General AI that also lets it do it's job effectively. It's not as simple as it looks. If you're not talking about a General AI than sure, it's easy, but non-General AI's are not very scary. > We already replace humans at their jobs all the time and they only very rarely kill their masters to prevent it. But humans don't have "maximize my production for this company" as their main life objective. Things like food, not going to jail, not dying, etc usually come first.