(Sample, over-simple failure mode: we try to make sure that our superintelligent machines work for the benefit of the human race by teaching them what happy and

(Sample, over-simple failure mode: we try to make sure that our superintelligent machines work for the benefit of the human race by teaching them what happy and unhappy people look like and telling them to make there be more happy people and fewer unhappy ones; they slaughter the entire human race and fill the universe with lots of little dolls that look just like really happy people.) A sample, less simple failure mode, also written by Yudkowsky: http://lesswrong.com/lw/xu/failed_utopia_42/ http://lesswrong.com/lw/xu/failed_utopia_42/