I think that OP is somewhat naive with some of these examples. > Or some idiot MBA type uses these AIs to optimise some business process and the AI decides dum

I think that OP is somewhat naive with some of these examples. > Or some idiot MBA type uses these AIs to optimise some business process and the AI decides dumping radioactive waste in the ocean is the best way to do it. The idea that a relatively simple AI (i.e. non-AGI) will have so much control over a system that it can get to the point where radioactive waste is being dumped feels outlandish to me. That would involve multiple systems and definitely some human beings. Just because an AI would theoretically make a harmful decision in an effort to optimize something doesn't that decision will come to pass - there are still going to be plenty of safeguards for things like this for the foreseeable future. Your Uber example is sort of relevant, but I don't think it's fair to say it's the same as cutting off power during a snowstorm. One of those things is much more highly regulated than the other (maybe with the exception of Texas and their power grid). The other thing to bear in mind here is that while it's not impossible these things are going to happen, the reality is they'll likely happen once and then face a huge backlash. Your Uber point is a good example of this - we no longer see surge pricing spiking during emergencies. Though again, I'll exempt the Texas power grid from that claim... it'll probably freeze over year after year because it'd be a travesty to regulate that sort of a critical utility. I could AI crashing it repeatedly for the same reason.