We just had a series of debates/discussions on this topic at my university, the results of which were pretty inconclusive. There are just too many possible sce

We just had a series of debates/discussions on this topic at my university, the results of which were pretty inconclusive. There are just too many possible scenarios which seem to require different responses, and in most cases to provide those responses is to answer philosophical questions that have been around for millennia. The strategies for mitigating risk seem to be: ensure that the AIs are controllable; avoid situations where there is a single AI (whether controlled or uncontrolled) that is too powerful; and ensure that the AI's goals are broadly acceptable to humankind. The first and the third objectives are extremely difficult, not just technically, but even from a conceptual standpoint[1]. The second strategy is reasonable, because even if a superhuman intelligence were somehow well controlled, depending on who controls it the outcomes could vary significantly. So perhaps the best thing we can hope for is something similar to society's current status quo-- lots of power concentrated in few hands[2], but without one single (person|corporation|government) being so dominant as to be able to act in opposition to all others. I am not confident that we will ever be able to produce a provably safe AI, or that we could get even a large majority of the world's population to agree on what a "good AI" might do without devolving into ineffectual generalities[3]. Supposing that resolving these questions is not prima facie impossible, it's not like retarding AI development comes without cost-- just about every facet of our lives can be improved via AI, and so in the years, decades, or centuries between when superhuman machine intelligence is theoretically achievable and the time when we collectively agree we can implement it safely, how many billions will suffer or die from things that we could've solved via AI[4]? On the whole, OpenAI sounds like a good idea. Making research broadly available helps avoid catastrophic "singleton" like futures, while accelerating the progress we make in the present. In addition, if there's every an AI SDK with effective methods of improving how "safe" a given AI is, most researchers would likely incorporate that into their work. It might not be "proven safe", but if there was a means to shut down a runaway process, or stop it from spreading to the Internet, or alert someone when it starts constructing androids shaped like Austrian bodybuilders, that would be handy. Responsible researchers should be doing this already, but as Scott points out the ones we should be worried about aren't responsible researchers. Open AI development is in harmony with safe AI development, at least in some respects. ------ 1. I have a significantly longer response that I scrapped because it might ultimately be better suited as a blog post or some such. 2. That's why it's called a power law distribution. Well, no, that's not it at all, but it seemed like a funny, flippant thing to say. 3. A universally beloved AI might be the equivalent of a Chinese Room where regardless of what message you send it, it responds with a vaguely complimentary yet motivational apothegm. 4. Bostrom tends to counterbalance this by arguing how much of our light cone (the "cosmic endowment") we might lose out on if we end up going extinct, due to, e.g., superhuman machine intelligence. Certainly "all of configurations of spacetime reachable from this point" outweighs the suffering of mere billions of people by some evaluations, but I ask myself "how much do I care about people thousands or millions of years into the future?", and also "if these guys have such a good handle on what constitutes the 'right' utility function, why haven't they shared it?". A more sarcastic variation of the above might be to remark that if they're able to approximate what people want with such high fidelity that they feel comfortable performing relativistic path integration over possible futures, then superintelligence is already here. ------