The scenario that inspired this game includes the contention that an AI could harm humanity without anything that we would recognize as malice in human terms.

The scenario that inspired this game includes the contention that an AI could harm humanity without anything that we would recognize as malice in human terms. https://intelligence.org/2013/05/05/five-theses-two-lemmas-and-a-couple-of-strategic-implications/ https://intelligence.org/2013/05/05/five-theses-two-lemmas-a... > For example, if you want to build a galaxy full of happy sentient beings, you will need matter and energy, and the same is also true if you want to make paperclips. This thesis is why we’re worried about very powerful entities even if they have no explicit dislike of us: “The AI does not love you, nor does it hate you, but you are made of atoms it can use for something else.” Note though that by the Orthogonality Thesis you can always have an agent which explicitly, terminally prefers not to do any particular thing — an AI which does love you will not want to break you apart for spare atoms. See: Omohundro (2008); Bostrom (2012). I guess two popular ways of disputing this thesis are to say that no AI that we can build can actually become this powerful, or to say that we'll easily program AIs not to harm us. I think the first objection is more cogent because there are various ways that you could argue that we're far from understanding how to build AIs of this kind. The second objection is a subtle problem; the Omohundro-Bostrom-Yudkowsky-Bostrom argument is very focused on the idea that there are so many hidden assumptions about what it means not to harm us, and what kinds of things are off-limits, that someone who tries to capture all of them algorithmically at one stroke is more likely to fail than to succeed. And quite a few of the toy game-playing AI systems have found ways to cheat by human standards and miss the essence of the human understanding of the task they were set. I think your analogy to the banknote-maximizer makes some of that subtlety clear: if we don't know what things are off-limits, we could see how a banknote-maximizer could cause very severe externalities without malice. (The earlier Omohundro argument points out that, whatever you want to accomplish, power and safety will help you accomplish it, so you always have a good reason to try to acquire them. Maybe an AI could be explicitly programmed not to want to acquire power and safety, but there, again, we'd want a very good way to describe exactly what those things do or don't consist of.)