> If easy instances are enough, what's the relevance of the provably ideal uncomputable problem, besides rhetoric? It shows that there's a path to AI that's ga

> If easy instances are enough, what's the relevance of the provably ideal uncomputable problem, besides rhetoric? It shows that there's a path to AI that's gated on solving formal math problems instead of imitating wishy-washy emotional humans. This point may be obvious to you, but apparently it wasn't obvious to Stross. > If humans used easy instances, and they're the only example we know of that achieved I, how is that not weak evidence that AI through easy instances will pass through human-like I? Humans are also the only animals that can play chess, but chessplaying software didn't pass through a human-like stage where they got nervous, forgetful, etc. My intuition says math can't be that radically different from chess (both involve search on trees with a complicated evaluation function), and we're only a handful of new insights from having computers decisively beat humans at math. And at that moment, by implication from my previous point, they might just also beat us at everything else :-) Remember that, on an absolute scale, humans are about as bad at math as you can be and still build a civilization (otherwise civilization would've happened at a lower level of development) and also that we probably suck at math because math is a conscious task (see Moravec's paradox). Related link that should make you scared if you buy my reasoning: http://www.vetta.org/2009/09/monte-carlo-aixi/ http://www.vetta.org/2009/09/monte-carlo-aixi/ .