More briefly, to respond to just what you said here: "It's that every intelligent machine we could build would be inherently dangerous." My view of the matter

More briefly, to respond to just what you said here: "It's that every intelligent machine we could build would be inherently dangerous." My view of the matter is that this is absolutely, 100% true. Any intelligent machine we build will eventually be used by someone that does not understand the serious risks involved in using it in various ways. It will be smart people doing this stuff, but people that haven't really considered the unknown dangers. The real problem is that with self-modifying AI (and trust me, even if it doesn't start out self-modifying, someone will try it eventually...), even seemingly innocuous things could be very dangerous - think of the difficult bullshit we go through to do fundamental physics experiments, and then imagine something 1000x smarter than us, but locked in a box... I can see the Hacker News headlines now: - Facebook loses 100k lines of AI code to disgruntled employee, Wikileaks puts it up for download - Show HN: I Hooked Facebook Brain up to my Roomba and taught it to do the Macarena! - Rate my startup: I set up a Hadoop cluster of Brain nodes working to program themselves in Haskell, and I'm calling it a business! - IRC Bot using Facebook Brain and Node.js passes the Turing Test in less than 20 lines of code [and yes, "Facebook Brain" is my personal vision of hell, too]