The whole point is that the model is bad. The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than

The whole point is that the model is bad. The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than giving someone a speck of dust in the eye". Actually following that rule could be disastrous (if the AI is powerful enough), because it will realize: * some people spontaneously torture others * the chances of spontaneous torture is a bit lower if all people are experiencing a barrage of dust specks sufficient to blind them * so: eyeball dust storms commencing in 3, 2, 1... One lesson to pick up here is that if you go by intuition, rather than doing the hard work of figuring out a logic way to decide this, if the hidden assumptions behind the intuition stop being true, we'll be in a bad place.