"We can make it impossible for the machine to override these drives." This has been debated to death, and nobody has come up with a good way to constrain a pro

"We can make it impossible for the machine to override these drives." This has been debated to death, and nobody has come up with a good way to constrain a program that's smarter than we are from rewriting itself "badly" (i.e. in a way that's bad for humans), either by accident or on purpose. All the "put it in a box", "don't give it a body", "don't let it on the Internet", "don't let it change itself" scenarios depend on all humans everywhere abiding by a set of rules, even though the AI-in-a-box might be actively trying to get people to let it out. There are many points of failure, and all it takes is one rogue research group somewhere to let one of them happen to end up with a self-modifying AI that starts to get smarter faster than we know how to stop it. Hell, if we're at the point where we can build an AI, one of the first things any smart asshole with a lot of compute power in his garage (or enough money to put together a massive EC2 cluster) will do is set the thing off on improving its own code, just to see what happens. And again, the problem is not that the computer will necessarily want to kill us or anything like that; it's that if it gets to the point where it's intelligence dwarfs ours, it will have such unbelievable power that even the slightest mistake could wipe us out. If we can build an AI that can improve itself, an AI that has self-improved for a while may be able to unleash any number of grey-goo style scenarios on the world. Might not be meant to kill us, but could happen by mistake. The "drives" that guide such an AI must specifically be aimed at avoiding those scenarios, including avoiding any self-modifications that would cause those scenarios. It's a much more difficult problem than it seems at first glance.