> A true superintelligence should be able to figure out what people are. Oh yes it would. How to exploit that fact into something that can reliably be carried

> A true superintelligence should be able to figure out what people are. Oh yes it would. How to exploit that fact into something that can reliably be carried out without misinterpretation however, is less clear. Imagine you're just telling the AI to "do what humanity wants you to". Hmm, what do we want? How individual wills can translate into collective (dis)agreement? How drives in a single individual translates into will? Depending on how the machine answers all those questions, it might perform quite differently. How about perpetuating a cycle of pain an hatred, on the basis that many people have (sometimes mortal) grudges? And even if you solve that, our current ideals may not be so ideal upon closer inspection (say, when our lives change and we take a second look). Would you want the AI freeze our current morality into an unwavering tradition? It's not clear it wouldn't. Much work to be done.