4 ms·

Your rebuttal can be used as is to argue that an oracle to the Halting Problem is a superintelligence, and that Stross's argument doesn't demonstrate the unlike

by anatoly 15y ago

Your rebuttal can be used as is to argue that an oracle to the Halting Problem is a superintelligence, and that Stross's argument doesn't demonstrate the unlikelihood of someone implementing a fast approximation to it tomorrow.

cousin_it 15y ago
You're right but this doesn't kill my argument, because not all instances of the halting problem are difficult. It's quite possible that some path to godhood (compared to a puny human) passes only through "easy" instances of the halting problem, efficiently solvable by some simple algorithm. In fact this is how we humans got to our current state of science and progress. Our math capabilities are evolutionarily new and inefficient: some of us can't even multiply 32-bit numbers!
- anatoly 15y ago
  Sure it's possible, but you have no evidence for it. Consider what your argument reduces to: - CS says the path to super-intelligent AI lies through human-equivalent AI (actually, he says Vinge's program says that, and he explicitly criticizes it) - but I can find an uncomputable perfectly ideal math function - and I can hypothesize that maybe it'll be possible to approximate it cheaply - and I can hypothesize that easy instances of it are good enough - and I will now view human progress as a way of approximating that function, and submit that only easy instances were used. This isn't merely red herring, it's red herring full of contradictions. If easy instances are enough, what's the relevance of the provably ideal uncomputable problem, besides rhetoric? If humans used easy instances, and they're the only example we know of that achieved I, how is that not weak evidence that AI through easy instances will pass through human-like I?
  - cousin_it 15y ago
    > If easy instances are enough, what's the relevance of the provably ideal uncomputable problem, besides rhetoric? It shows that there's a path to AI that's gated on solving formal math problems instead of imitating wishy-washy emotional humans. This point may be obvious to you, but apparently it wasn't obvious to Stross. > If humans used easy instances, and they're the only example we know of that achieved I, how is that not weak evidence that AI through easy instances will pass through human-like I? Humans are also the only animals that can play chess, but chessplaying software didn't pass through a human-like stage where they got nervous, forgetful, etc. My intuition says math can't be that radically different from chess (both involve search on trees with a complicated evaluation function), and we're only a handful of new insights from having computers decisively beat humans at math. And at that moment, by implication from my previous point, they might just also beat us at everything else :-) Remember that, on an absolute scale, humans are about as bad at math as you can be and still build a civilization (otherwise civilization would've happened at a lower level of development) and also that we probably suck at math because math is a conscious task (see Moravec's paradox). Related link that should make you scared if you buy my reasoning: http://www.vetta.org/2009/09/monte-carlo-aixi/ http://www.vetta.org/2009/09/monte-carlo-aixi/ .

by anatoly 15y ago

cousin_it 15y ago

You're right but this doesn't kill my argument, because not all instances of the halting problem are difficult. It's quite possible that some path to godhood (compared to a puny human) passes only through "easy" instances of the halting problem, efficiently solvable by some simple algorithm. In fact this is how we humans got to our current state of science and progress. Our math capabilities are evolutionarily new and inefficient: some of us can't even multiply 32-bit numbers!

anatoly 15y ago
Sure it's possible, but you have no evidence for it. Consider what your argument reduces to: - CS says the path to super-intelligent AI lies through human-equivalent AI (actually, he says Vinge's program says that, and he explicitly criticizes it) - but I can find an uncomputable perfectly ideal math function - and I can hypothesize that maybe it'll be possible to approximate it cheaply - and I can hypothesize that easy instances of it are good enough - and I will now view human progress as a way of approximating that function, and submit that only easy instances were used. This isn't merely red herring, it's red herring full of contradictions. If easy instances are enough, what's the relevance of the provably ideal uncomputable problem, besides rhetoric? If humans used easy instances, and they're the only example we know of that achieved I, how is that not weak evidence that AI through easy instances will pass through human-like I?
- cousin_it 15y ago
  > If easy instances are enough, what's the relevance of the provably ideal uncomputable problem, besides rhetoric? It shows that there's a path to AI that's gated on solving formal math problems instead of imitating wishy-washy emotional humans. This point may be obvious to you, but apparently it wasn't obvious to Stross. > If humans used easy instances, and they're the only example we know of that achieved I, how is that not weak evidence that AI through easy instances will pass through human-like I? Humans are also the only animals that can play chess, but chessplaying software didn't pass through a human-like stage where they got nervous, forgetful, etc. My intuition says math can't be that radically different from chess (both involve search on trees with a complicated evaluation function), and we're only a handful of new insights from having computers decisively beat humans at math. And at that moment, by implication from my previous point, they might just also beat us at everything else :-) Remember that, on an absolute scale, humans are about as bad at math as you can be and still build a civilization (otherwise civilization would've happened at a lower level of development) and also that we probably suck at math because math is a conscious task (see Moravec's paradox). Related link that should make you scared if you buy my reasoning: http://www.vetta.org/2009/09/monte-carlo-aixi/ http://www.vetta.org/2009/09/monte-carlo-aixi/ .