Another example, even more bare-bones, is AI. We now have programs that "understand" certain domains better than humans. And yet there are no "simple terms" to

Another example, even more bare-bones, is AI. We now have programs that "understand" certain domains better than humans. And yet there are no "simple terms" to explain their understanding, reduce it to a neat closed-form formula. Isn't it a mistake to assume a nice, elegant explanation, a cute human "narrative", must always exist? "But Radim, the computers don't really understand the domain!". Sorry, a cop out, No True Scotsman fallacy. In what way do they not? This is crystal clear with Go: the little stories humans have built to deepen their understanding of the game over 2,000 years are irrelevant to a game's outcome. The details and variations are all that matters, and there is no way to reach the computer's level of understanding of the game without understanding the details.