4 ms·

Strongly disagree. Shapley values and LIME give a very crude and extremely limited understanding of the model. They basically amount to a random selection of lo

Strongly disagree. Shapley values and LIME give a very crude and extremely limited understanding of the model. They basically amount to a random selection of local slopes. For instance, if I tell you the (randomly selected) slopes of a function are [0.5, 0.2, 12.4, 1.1, 2.6] which the average is 3.3, can you guess anything? You might notice it is monotonic (maybe) but you certainly don't guess it is e^x.

LeanderK 5y ago
I would say that our ML models are not predictable enough yet at the local neighbourhood to really trust LIME. Adversarial examples prove that you just can't select a small enough range since you can always find those even for super tiny distances.
oli5679 5y ago
It depends on your domain. If you have a relatively small number of features and your features have relatively stable but not necessarily linear partial dependence, then shap can unpack them pretty well. E.g. value of a house it might not necessarily increase linearly in squarefoot, distance from city centre and other factors so a random-forest/gbt might do better than a linear model without careful feature engineering. Shap is a brilliant tool to unpack the dependencies this model has learned. There will be other cases where a linear model is adequate, so you can get the explainability just from the coefficients, or where there are very large numbers of features with complicated nested structure. In either case shap is not too useful. But the 'fit lightgbm and explain it with shap' is a decently pragmatic approach for lots of tabular data machine-learning problems.
oneoff786 5y ago
Maybe. But I don’t care about estimating the slopes. I just need to know big values of feature X pull the model prediction up and Small values do nothing. If you really need to unpack it, cluster your shaps. To see what kinds of predictions are being made

LeanderK 5y ago

I would say that our ML models are not predictable enough yet at the local neighbourhood to really trust LIME. Adversarial examples prove that you just can't select a small enough range since you can always find those even for super tiny distances.

oli5679 5y ago

It depends on your domain. If you have a relatively small number of features and your features have relatively stable but not necessarily linear partial dependence, then shap can unpack them pretty well. E.g. value of a house it might not necessarily increase linearly in squarefoot, distance from city centre and other factors so a random-forest/gbt might do better than a linear model without careful feature engineering. Shap is a brilliant tool to unpack the dependencies this model has learned. There will be other cases where a linear model is adequate, so you can get the explainability just from the coefficients, or where there are very large numbers of features with complicated nested structure. In either case shap is not too useful. But the 'fit lightgbm and explain it with shap' is a decently pragmatic approach for lots of tabular data machine-learning problems.

oneoff786 5y ago

Maybe. But I don’t care about estimating the slopes. I just need to know big values of feature X pull the model prediction up and Small values do nothing. If you really need to unpack it, cluster your shaps. To see what kinds of predictions are being made