4 ms·

This needs more emphasis on the "correlation is NOT necessarily causation" part. In my experience trying to use SHAP and other "interpretable ML" techniques for

This needs more emphasis on the "correlation is NOT necessarily causation" part. In my experience trying to use SHAP and other "interpretable ML" techniques for data science analysis, trying to identify relevant drivers for a metric based on SHAP-like techniques is a recipe for disaster. You'll get results that look on the surface to someone that is well versed but not an expert in the business like they're legit but to an expert they're obvious garbage. The way I and many others were taught in school is that you get some dataset and you throw it into a predictor and get results, but what they should've taught is how to find things (among hundreds of possible attributes, or even attributes that aren't collected yet) you think would be causal predictors first without looking at the data and then explore and build a model based on that.

aidancooper 5y ago
Hi, author here. I agree and I tried to address this in the passage pasted below, but you're right that this deserves emphasis. I hope the reference to Scott Lundberg's excellent article on SHAP and causality provides the additional context that was out of scope for this primer. I would add, however, that even if a relationship isn't causative, it's still potentially insightful to understand which variables are influencing an ML model's predictions. --- Caution should be exercised when deriving insights from SHAP analyses. It is important not to become over-invested in conclusions that certain features cause certain outcomes unless the experiment has been conducted within a causal framework (this is rarely the case).[6] SHAP only tells you what the model is doing within the context of the data on which it has been trained: it doesn't necessarily reveal the true relationship between variables and outcomes in the real world. Decision-makers are often tempted to view features in SHAP analyses as dials that can be manipulated to engineer specific outcomes, so this distinction must be communicated.

aidancooper 5y ago

Hi, author here. I agree and I tried to address this in the passage pasted below, but you're right that this deserves emphasis. I hope the reference to Scott Lundberg's excellent article on SHAP and causality provides the additional context that was out of scope for this primer. I would add, however, that even if a relationship isn't causative, it's still potentially insightful to understand which variables are influencing an ML model's predictions. --- Caution should be exercised when deriving insights from SHAP analyses. It is important not to become over-invested in conclusions that certain features cause certain outcomes unless the experiment has been conducted within a causal framework (this is rarely the case).[6] SHAP only tells you what the model is doing within the context of the data on which it has been trained: it doesn't necessarily reveal the true relationship between variables and outcomes in the real world. Decision-makers are often tempted to view features in SHAP analyses as dials that can be manipulated to engineer specific outcomes, so this distinction must be communicated.