3 ms·

What I see here is proof that this is a bad model that produces insane results, rather than it proving that I've made a bad decision.

• jtheory 12y ago
  The whole point is that the model is bad. The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than giving someone a speck of dust in the eye". Actually following that rule could be disastrous (if the AI is powerful enough), because it will realize: * some people spontaneously torture others * the chances of spontaneous torture is a bit lower if all people are experiencing a barrage of dust specks sufficient to blind them * so: eyeball dust storms commencing in 3, 2, 1... One lesson to pick up here is that if you go by intuition, rather than doing the hard work of figuring out a logic way to decide this, if the hidden assumptions behind the intuition stop being true, we'll be in a bad place.
  • zyxley 12y ago
    > The whole point is that the model is bad. That doesn't seem to be Yudkowsky's argument.