The Go book The Treasure Chest Enigma by Noriyuki Nakayama starts with an essay that gives the book its title. The Treasure Chest Enigma is a life-and-death pro

The Go book The Treasure Chest Enigma by Noriyuki Nakayama starts with an essay that gives the book its title. The Treasure Chest Enigma is a life-and-death problem too hard for professional Go players, but easy for amateurs. A real life problem that caused much consternation and merriment in the Nihon Ki-in in 1958. It relies on the technique that is at the heart of this paper. There is a small dead group completely surrounded by a larger group. But the larger group runs out of liberties and the small dead group comes back to life when the big group is captured. This kind of thing never comes up in professional play. (The game that it came from was part of the qualifying tournament for professional shodan; Mizuno Hiroshi and Noriyuki Nakayama were not yet professional players.) All the top players had drilled so hard on the life and death problems that actually arise in professional play that they had developed a peculiar blind spot around capturing races with one group completely surrounded by another. My expectation is that the first fifty years of Artificial Intelligence disasters will be dominated by humans being over-impressed by the intelligence of AI systems and giving them responsibilities that they are not read for. This article https://www.gbnews.com/news/ai-latest-news-judge-legal-decisions https://www.gbnews.com/news/ai-latest-news-judge-legal-decis... already speculating that "AI machines 'could soon make legally binding decisions' in UK, says senior judge" is the kind of thing I fear. I don't know what to make of The Treasure Chest Enigma. Theory one: it tells us that we are already coping successfully with this problem in human intelligence. Yes, AI has weird mental blind spots, but people do too and there is no need to worry. Theory two: Human mental blind spots are smaller and more obscure that the mental blind spots of current AI. There is plenty of quantitative difference so that we can legitimately worry that AI blind spots will cause unexpected disasters. We will be taken by surprise because we are used to human intelligence with its small, benign blind spots. The significance of The Treasure Chest Enigma is that it exists, which tells us that mental blind spots are inherent to intelligence. We will not solve the problem of AI having large, dangerous mental blind spots easily. We can only hope to shrink the problem down to an ineradicable core.