It isn't likely to work like that. The much more plausible outcome is that we'll choose to let AI take over, to optimise for a certain subset of possibilities t

It isn't likely to work like that. The much more plausible outcome is that we'll choose to let AI take over, to optimise for a certain subset of possibilities that we deem important. It's like what happened with surveillance. I'm very confident that the smartphone I'm typing this message on can track my location, monitor who I'm in contact with and what I'm sharing with them, and all that information can be tracked by government agencies. You'd be a fool to expect otherwise at this point. Yet despite the power for this to be used against us (blackmail, censorship, etc...) we still carry on as we did before. Once AI becomes as integral to modern society as the Internet, you really think we'll be switching it off as soon as it misbehaves? Consider, what level of catastrophe would it take before we switched off the Internet?