A bit of background may be in order: the main point of this story (aside from just being a good story) is to get readers to think about how one ought to go abou

A bit of background may be in order: the main point of this story (aside from just being a good story) is to get readers to think about how one ought to go about interacting with other intelligent agents whose value system is radically different from one's own. See, e.g., http://lesswrong.com/lw/tn/the_true_prisoners_dilemma/ http://lesswrong.com/lw/tn/the_true_prisoners_dilemma/ for an explanation of why those radically different values make a difference. And a bit of background to that bit of background: Yudkowsky is interested in the question "Suppose it turns out, in the not-outrageously-distant future, to make machines whose general-purpose intelligence matches or exceeds our own; and suppose it turns out that once you've got that, ordinary technological progress plus the ability of those machines to do their own machine-designing leads rapidly to machines vastly smarter and more powerful than we are. How do we avoid this scenario playing out in a way that we don't find abhorrent?" (Sample, over-simple failure mode: we try to make sure that our superintelligent machines work for the benefit of the human race by teaching them what happy and unhappy people look like and telling them to make there be more happy people and fewer unhappy ones; they slaughter the entire human race and fill the universe with lots of little dolls that look just like really happy people.) Well, that's at least partly a question about how to deal with clashes of values between very different sorts of intelligent agent: in this case, human beings and superintelligent AI machines.