We understand how people work and think pretty well, their edgecases, and countermeasures to said edgecases. We don't understand edge cases of AI much at all,

We understand how people work and think pretty well, their edgecases, and countermeasures to said edgecases. We don't understand edge cases of AI much at all, and the terrain-space of "possible minds" they might be is vastly broader than that of a human mind, and utterly outside our experience. An AI might work with absolute 100% flawless precision for 100 years, then decided one day to kill everyone. A lot harder to see that coming from a totally alien intelligence than a human one. AI does not think like humans, have human values, or care about what humans care about, that might make them extremely unpredictable and dangerous. There will be no time for iterative development, or learning from our mistakes, a sufficiently advanced AI knows we would try to stop it, and make sure that we never can. https://www.youtube.com/watch?v=tcdVC4e6EV4 https://www.youtube.com/watch?v=tcdVC4e6EV4