There is something close to what you're describing: 1) You train a complex ML model (say, deep neural net) to get the job done, solve your problem with high ac

There is something close to what you're describing: 1) You train a complex ML model (say, deep neural net) to get the job done, solve your problem with high accuracy. 2) When you want to explain one of your model's results, you train a simpler more explainable model (say, linear regression) in the neighborhood of that point, such that it is locally similar to the more complex model. The task is too complex to be explained by a linear regression model over all possible inputs, but it often is simple enough around a specific point.