If we're really careful about how we program the AI, then we might be able to make it shut down after doing some limited-scope task. However, if its task is not

If we're really careful about how we program the AI, then we might be able to make it shut down after doing some limited-scope task. However, if its task is not extremely limited in scope, then it has strong incentives to continue itself so that it can complete its task. An idea of how things could potentially go wrong if you're not ever so careful in this way is the following thought experiment: > You told the AI to make exactly 100 paperclips. But the AI is made of (and knows that it is made of) human-made components, which are notoriously fairly-to-mostly reliable but not 100% reliable. Say they're 99.999% reliable. The AI's sensors are currently giving it the answer of "I have made 100 paperclips". Can it trust them? It really really wants to have made 100 paperclips, and it's not completely sure of having made 100 paperclips, so it decides to verify whether it has made 100 paperclips. Boom, suddenly you have unbounded behaviour: "verify a probability". If the AI is not extremely carefully designed, unbounded behaviour will lead to the AI acquiring resources and perhaps attempting to improve itself, in order to better fulfill its task. Look how readily humans attempt to learn new things and get better at what we do, even though our own goal systems are extremely fuzzy: we don't pursue one task with single-minded devotion like we hope an AI would. (If our AI's goal systems are fuzzy, then we won't understand why it's doing what it's doing, and then it's probably curtains for humanity.)