4 ms·

The doomsday cult of the supermind-AI tends to think the "intelligence" is the solution to infinite power, so once the computer starts improving itself, humanit

The doomsday cult of the supermind-AI tends to think the "intelligence" is the solution to infinite power, so once the computer starts improving itself, humanity is at its mercy — as if better software alone could deliver the singularity, unchecked by material constraints. I suspect some of this comes from software-oriented nerds who think a slightly too much of the powers intelligence grants them.

jimbokun 5y ago
Doesn't have to be "unchecked", just much less checked than human brains. Although, that is an interesting question in and of itself. Are living brains near a local optimum in terms of computing power, given heat and power constraints? With Moore's Law, the prediction was that computer intelligence would surpass human intelligence in a few more doublings. But now that the primary constraint is more about power usage than absolute clock speed, maybe living brains compare better?
cogman10 5y ago
It comes from the false notion that infinite means unbounded. I see this debate come up all the time in the discussion of something like pi. You'll see the assertion "Oh, because pi is infinitely non-repeating, it must contain the complete works of Shakespeare" Now, does it? Maybe. However, just because something doesn't repeat, doesn't mean it has to cover all possible combinations. For example, you could have a number that follows the pattern 0.01001000100001000001.... Non repeating and infinite but contains nothing but a simple pattern that will never contain anything meaningful. AI is much the same. We think "all problems have a solution! So a super intelligence must know all the answers!" The assumption "all problems have a solution" is just an assumption, not a truth. FTL travel may very well be an impossibility. No amount of knowledge could reverse that possibility.

jimbokun 5y ago

Doesn't have to be "unchecked", just much less checked than human brains. Although, that is an interesting question in and of itself. Are living brains near a local optimum in terms of computing power, given heat and power constraints? With Moore's Law, the prediction was that computer intelligence would surpass human intelligence in a few more doublings. But now that the primary constraint is more about power usage than absolute clock speed, maybe living brains compare better?

cogman10 5y ago

It comes from the false notion that infinite means unbounded. I see this debate come up all the time in the discussion of something like pi. You'll see the assertion "Oh, because pi is infinitely non-repeating, it must contain the complete works of Shakespeare" Now, does it? Maybe. However, just because something doesn't repeat, doesn't mean it has to cover all possible combinations. For example, you could have a number that follows the pattern 0.01001000100001000001.... Non repeating and infinite but contains nothing but a simple pattern that will never contain anything meaningful. AI is much the same. We think "all problems have a solution! So a super intelligence must know all the answers!" The assumption "all problems have a solution" is just an assumption, not a truth. FTL travel may very well be an impossibility. No amount of knowledge could reverse that possibility.