> The other side of the argument seems to hold very odd assumptions about things like thermodynamics and robotics work, or even intelligence itself. There are

> The other side of the argument seems to hold very odd assumptions about things like thermodynamics and robotics work, or even intelligence itself. There are two major types of arguments for why we should be cautious about ultra-smart AI: 1. "Ultra-smart AI will rapidly improve its own abilities to vastly super-human intelligence, invent nanotech, and kill us all. Maybe in a week or two." This argument does require a number of very specific assumptions to be true. 2. "If humans are the second-smartest species on the planet, we may not get to make all the important decisions anymore. We might not enjoy this." (2) doesn't require any exotic ideas about thermodynamics or nanotech. It does assume we might be able to build something that's noticeably smarter than we are. Honestly, even if the best AI we could be build was equivalent to a very smart human (say, an IQ of 145, or three standard deviations from average), and if that AI was tireless and easily copyable, then the future would become very weird. Personally, I give us reasonable odds of us making it this far within my normal lifetime. And it might be possible to build something smarter than that, given time, depending on how "intelligence" works.