There might be some methods that could be considered using AI to figure our how the model works, like distillation or I think I've heard of simultaneously train

There might be some methods that could be considered using AI to figure our how the model works, like distillation or I think I've heard of simultaneously training feature maps along with a classifier. Those don't get around the fact that placing constraints on how the model can make its decision (by requiring that they have human interpretable explanations) limits the kinds of model that are available, and can limit performance. So the rule is more a slightly more advanced analogy of telling people they need to make a rubric with a list of factors, and then a weighing for each, and use that to make credit decisions.