If you have no idea what the superintelligence will do, then you should set your expectation to be the outcome of any random one of possible superintelligent mi

If you have no idea what the superintelligence will do, then you should set your expectation to be the outcome of any random one of possible superintelligent mind designs. Not all superintelligent mind designs are totally unpredictable -- if I know it will prefer winning in a game of chess to losing, I predict it will win every match against a human, even if I don't know exactly what moves it will make. If I know it will prefer existing to not existing, I predict it will seek the resources to continue existing. If the superintelligence has the hardwired goal of producing paperclips, one of the possible outcomes is tiling the solar system with paperclips. Another is that humans succeed in shutting it down but not until after it's killed people. Once you're done enumerating as many possibilities as you can for what a superintelligence might have as its goals, simple and complicated or even nonexistent, and what the various outcomes of a superintelligent agent with those goals broadly look like, find out how many of them are positive, negative, or neutral to the future of humanity, especially if these things the superintelligence does requires resources humans also use. I did this once, it convinced me that the likely case is something negative for humanity, even if it's unlikely to be a Terminator/Matrix/Hollywood scenario or really any specific enumeration. What further convinced me that the likely outcome is still negative for humanity, in the event of people making an honest but uninformed attempt at after having created an AGI making sure it won't be net-negative for humanity, is understanding the complexity of human value and how almost right is still very wrong. (http://wiki.lesswrong.com/wiki/Complexity_of_value http://wiki.lesswrong.com/wiki/Complexity_of_value)