4 ms·

> Most fast-takeoff AGI scenarios usually involve a singular intelligence created from an algorithm that easily scales based on how much computing power is allo

by baddox 11y ago

> Most fast-takeoff AGI scenarios usually involve a singular intelligence created from an algorithm that easily scales based on how much computing power is allocated to it. Still, it's pretty easy to imagine this not resulting in the sort of "runaway" intelligence most often associated with the notion of singularity. What if general intelligence is inherently extremely computationally complex? Imagine that the level of intelligence is represented by a number n, and say that human intelligence is n=100. If the computational complexity of intelligence is, say, O(1000^n), we won't expect to see hardware that's twice as intelligent as humans any time soon even if we know the algorithm.

rl3 11y ago
If that were the case, I would agree we would be quite far away from simulated human-level cognition. My main point was that either we have the computational capacity, or we do not. If we do, the chances of overshooting the human-level mark by a large margin are quite high, hence the risk. Another angle on this is that perhaps human cognition has an unnecessarily high degree of computational complexity, a degree that's far above the threshold required for an AGI to be dangerous. The Paperclip Maximizer[0] is an excellent example of this. Perhaps due to low computational complexity of its architecture, the thing wouldn't even be sentient in any sense of the word, and yet it'd still be perfectly capable of destroying the world. Why? Because its initial intelligence would be determined by the amount of computing power allocated to it. Allocate enough computing power, and assuming you've passed the magic threshold, it would start allocating itself computing power in short order. Perhaps what we should fear most is not sentient AGI capable of thought similar to humans, but AGI that's a structured amalgamation of existing AI technology—if only because the latter is far more likely to be feasible first. [0] http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer

by baddox 11y ago

rl3 11y ago

If that were the case, I would agree we would be quite far away from simulated human-level cognition. My main point was that either we have the computational capacity, or we do not. If we do, the chances of overshooting the human-level mark by a large margin are quite high, hence the risk. Another angle on this is that perhaps human cognition has an unnecessarily high degree of computational complexity, a degree that's far above the threshold required for an AGI to be dangerous. The Paperclip Maximizer[0] is an excellent example of this. Perhaps due to low computational complexity of its architecture, the thing wouldn't even be sentient in any sense of the word, and yet it'd still be perfectly capable of destroying the world. Why? Because its initial intelligence would be determined by the amount of computing power allocated to it. Allocate enough computing power, and assuming you've passed the magic threshold, it would start allocating itself computing power in short order. Perhaps what we should fear most is not sentient AGI capable of thought similar to humans, but AGI that's a structured amalgamation of existing AI technology—if only because the latter is far more likely to be feasible first. [0] http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer