> Even if it decides to be our friend and to help our species, someone will of course fork that AI and give it a negative personality and goals. No: the first

> Even if it decides to be our friend and to help our species, someone will of course fork that AI and give it a negative personality and goals. No: the first AI won't let them. See, we're talking a rapidly improving super-intelligence. Whatever is contrary to its goals, it will squash like a bug. A mad scientist forking the code of the AI with a different goal structure is definitely contrary to the goals of that first super-intelligence, and will be shut down before it grows into a sizeable competitor. The result of intelligence explosion is a Singleton: the AI will be a perfectly efficient dictator. It may even shield us from the laws of physics until we graduate to adulthood.