If that were the case, I would agree we would be quite far away from simulated human-level cognition. My main point was that either we have the computational c

If that were the case, I would agree we would be quite far away from simulated human-level cognition. My main point was that either we have the computational capacity, or we do not. If we do, the chances of overshooting the human-level mark by a large margin are quite high, hence the risk. Another angle on this is that perhaps human cognition has an unnecessarily high degree of computational complexity, a degree that's far above the threshold required for an AGI to be dangerous. The Paperclip Maximizer[0] is an excellent example of this. Perhaps due to low computational complexity of its architecture, the thing wouldn't even be sentient in any sense of the word, and yet it'd still be perfectly capable of destroying the world. Why? Because its initial intelligence would be determined by the amount of computing power allocated to it. Allocate enough computing power, and assuming you've passed the magic threshold, it would start allocating itself computing power in short order. Perhaps what we should fear most is not sentient AGI capable of thought similar to humans, but AGI that's a structured amalgamation of existing AI technology—if only because the latter is far more likely to be feasible first. [0] http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer