I like the metaphor, but I think it would be more appropriate to flip the roles: it’s more akin to asking the person using the models (or the airplane in your s

I like the metaphor, but I think it would be more appropriate to flip the roles: it’s more akin to asking the person using the models (or the airplane in your simile, the pilot) to understand how their work (i.e. have basic mechanical understanding). As it happens, that’s precisely what we ask pilot: they need to have enough mechanical engineering talent to inspect the plane in detail before take of as well as understand, diagnose and fix a problem in the air. To come back to modelling: I see daily people with a cursory understanding of models misusing their tools and not noticing blatant problems. Just today: - someone using random forest to predict a quantity (which you can do, it’s often nicknamed “random tree regression”) so the first thing their did was to turn quantities to predict to as many categorical variables; - someone else, on the same problem didn’t see why having a massively higher mean square error on the out-of-bag sample compared to the test sample was suspicious. I’m not saying that five years of university will solve that, but I’m noticed that taking the time to dig into the model, vs. assuming it’s like a car -- you only need to turn the key and go with it -- is strongly correlated with making those mistakes.