Is it maybe possible to combine the black box models with more explainable ones in a reasonable way? Use a good black box model to do the actually prediction, a

Is it maybe possible to combine the black box models with more explainable ones in a reasonable way? Use a good black box model to do the actually prediction, and then consult some more explainable models for a(notice that I use "a" not "the") explanation. One naive approach would be to have an ensamble of explainable models, and pick only those which gives the same answer as the black box model, and then use the explanation from them? I guess it depends on what one wants to use the explanation for. Usually you want to deduce some actions you can do to change stuff in the real world, and with an explainable model you hope that the model has captured some truth which translates to the real world, and with the very naive approach above you hope that the good black box model helps us pick the explainable model which had best understood the world (or at least the world according to the precise parameters it has been run with now). Maybe one can try some thousand permutations of the input all around the original input(into the black box model) , and use that to find the explainable model which had best understood the solution space just around there, and extract explanations from it?