AGI is like the nuclear bomb in more ways than one. It has the capacity to end the world, or start research that ends up making the world a utopia. It's also in

AGI is like the nuclear bomb in more ways than one. It has the capacity to end the world, or start research that ends up making the world a utopia. It's also inevitable like the nuclear bomb -- no international treaty could have stopped the Manhattan Project, because it was seen as a moral imperative to those involved that they continue their research and create the bomb before someone else did. The same principle applies to AGI. If principled people don't invent AGI first, that means unprincipled people will, which could spend the end of the entire world.