4 ms·

I am disappointed by the fact that people are so willing to throw model explainability out of the window because black-box models have higher predictive accurac

I am disappointed by the fact that people are so willing to throw model explainability out of the window because black-box models have higher predictive accuracy and sound cooler. Or they say that SHAP can solve interpretability even though the model that SHAP presents is not the same as the one that is making the predictions. This can hide biases or reliability problems. I think throwing a big (semi) black-box at the problem is short term thinking. In this maintenance context, for example, you can only improve airplanes if you know WHY you need to do maintenance. So therefore you need explainable models. Put differently, you need explainable models to learn from the model and to steer it based on knowledge that domain experts have. We’ve learned the importance of this when working with the Dutch military on predicting who would drop out from special forces selection [1]. And yes, we had a smaller dataset, but I don’t think that matters so much. It takes effort to get an explainable model working on the data, but the knowledge that you obtain compounds and gives you better models over time. [1]: https://doi.org/10.31234/osf.io/s6j3r https://doi.org/10.31234/osf.io/s6j3r

huijzer 3y ago
And sure, throw black-box models at low-stake decisions like recommender systems, sentiment classification, or spam filtering, but I don’t think it’s wise for high-stake decisions where the decisions cannot easily be verified by a human. But well, it doesn’t matter what’s the truth here. C3.ai wants to sell fancy algorithms and some Lt. Col. at the military wants to sound innovative. They both succeeded.
- ChatGTP 3y ago
  This
Epa095 3y ago
Is it maybe possible to combine the black box models with more explainable ones in a reasonable way? Use a good black box model to do the actually prediction, and then consult some more explainable models for a(notice that I use "a" not "the") explanation. One naive approach would be to have an ensamble of explainable models, and pick only those which gives the same answer as the black box model, and then use the explanation from them? I guess it depends on what one wants to use the explanation for. Usually you want to deduce some actions you can do to change stuff in the real world, and with an explainable model you hope that the model has captured some truth which translates to the real world, and with the very naive approach above you hope that the good black box model helps us pick the explainable model which had best understood the world (or at least the world according to the precise parameters it has been run with now). Maybe one can try some thousand permutations of the input all around the original input(into the black box model) , and use that to find the explainable model which had best understood the solution space just around there, and extract explanations from it?

huijzer 3y ago

And sure, throw black-box models at low-stake decisions like recommender systems, sentiment classification, or spam filtering, but I don’t think it’s wise for high-stake decisions where the decisions cannot easily be verified by a human. But well, it doesn’t matter what’s the truth here. C3.ai wants to sell fancy algorithms and some Lt. Col. at the military wants to sound innovative. They both succeeded.

ChatGTP 3y ago
This

Epa095 3y ago

Is it maybe possible to combine the black box models with more explainable ones in a reasonable way? Use a good black box model to do the actually prediction, and then consult some more explainable models for a(notice that I use "a" not "the") explanation. One naive approach would be to have an ensamble of explainable models, and pick only those which gives the same answer as the black box model, and then use the explanation from them? I guess it depends on what one wants to use the explanation for. Usually you want to deduce some actions you can do to change stuff in the real world, and with an explainable model you hope that the model has captured some truth which translates to the real world, and with the very naive approach above you hope that the good black box model helps us pick the explainable model which had best understood the world (or at least the world according to the precise parameters it has been run with now). Maybe one can try some thousand permutations of the input all around the original input(into the black box model) , and use that to find the explainable model which had best understood the solution space just around there, and extract explanations from it?