How would you debug such a system? E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which lead

How would you debug such a system? E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations. If you have no idea how the program arrives at its results or even if the results are correct or not, how would you be able to find that bug? Similarly, how do you make sure the program doesn't learn garbage features or overfits on your testing environment? There are enough war stories of image classifiers that just learned some subtle lighting differences in the trainset photos of something that actually distinguished the object in question. Or how do you prevent models learning certain features that you specifically don't want to learn, such as skin colour? I think developing those models without trying to understand what they do has a high risk of leading to magical thinking.