4 ms·

Sure it's possible, but you have no evidence for it. Consider what your argument reduces to: - CS says the path to super-intelligent AI lies through human-equi

by anatoly 15y ago

Sure it's possible, but you have no evidence for it. Consider what your argument reduces to: - CS says the path to super-intelligent AI lies through human-equivalent AI (actually, he says Vinge's program says that, and he explicitly criticizes it) - but I can find an uncomputable perfectly ideal math function - and I can hypothesize that maybe it'll be possible to approximate it cheaply - and I can hypothesize that easy instances of it are good enough - and I will now view human progress as a way of approximating that function, and submit that only easy instances were used. This isn't merely red herring, it's red herring full of contradictions. If easy instances are enough, what's the relevance of the provably ideal uncomputable problem, besides rhetoric? If humans used easy instances, and they're the only example we know of that achieved I, how is that not weak evidence that AI through easy instances will pass through human-like I?

cousin_it 15y ago
> If easy instances are enough, what's the relevance of the provably ideal uncomputable problem, besides rhetoric? It shows that there's a path to AI that's gated on solving formal math problems instead of imitating wishy-washy emotional humans. This point may be obvious to you, but apparently it wasn't obvious to Stross. > If humans used easy instances, and they're the only example we know of that achieved I, how is that not weak evidence that AI through easy instances will pass through human-like I? Humans are also the only animals that can play chess, but chessplaying software didn't pass through a human-like stage where they got nervous, forgetful, etc. My intuition says math can't be that radically different from chess (both involve search on trees with a complicated evaluation function), and we're only a handful of new insights from having computers decisively beat humans at math. And at that moment, by implication from my previous point, they might just also beat us at everything else :-) Remember that, on an absolute scale, humans are about as bad at math as you can be and still build a civilization (otherwise civilization would've happened at a lower level of development) and also that we probably suck at math because math is a conscious task (see Moravec's paradox). Related link that should make you scared if you buy my reasoning: http://www.vetta.org/2009/09/monte-carlo-aixi/ http://www.vetta.org/2009/09/monte-carlo-aixi/ .

by anatoly 15y ago

cousin_it 15y ago

> If easy instances are enough, what's the relevance of the provably ideal uncomputable problem, besides rhetoric? It shows that there's a path to AI that's gated on solving formal math problems instead of imitating wishy-washy emotional humans. This point may be obvious to you, but apparently it wasn't obvious to Stross. > If humans used easy instances, and they're the only example we know of that achieved I, how is that not weak evidence that AI through easy instances will pass through human-like I? Humans are also the only animals that can play chess, but chessplaying software didn't pass through a human-like stage where they got nervous, forgetful, etc. My intuition says math can't be that radically different from chess (both involve search on trees with a complicated evaluation function), and we're only a handful of new insights from having computers decisively beat humans at math. And at that moment, by implication from my previous point, they might just also beat us at everything else :-) Remember that, on an absolute scale, humans are about as bad at math as you can be and still build a civilization (otherwise civilization would've happened at a lower level of development) and also that we probably suck at math because math is a conscious task (see Moravec's paradox). Related link that should make you scared if you buy my reasoning: http://www.vetta.org/2009/09/monte-carlo-aixi/ http://www.vetta.org/2009/09/monte-carlo-aixi/ .