3 ms·
Hi, author here. I agree and I tried to address this in the passage pasted below, but you're right that this deserves emphasis. I hope the reference to Scott Lu
by aidancooper 5y ago
Hi, author here. I agree and I tried to address this in the passage pasted below, but you're right that this deserves emphasis. I hope the reference to Scott Lundberg's excellent article on SHAP and causality provides the additional context that was out of scope for this primer. I would add, however, that even if a relationship isn't causative, it's still potentially insightful to understand which variables are influencing an ML model's predictions.
---
Caution should be exercised when deriving insights from SHAP analyses. It is important not to become over-invested in conclusions that certain features cause certain outcomes unless the experiment has been conducted within a causal framework (this is rarely the case).[6] SHAP only tells you what the model is doing within the context of the data on which it has been trained: it doesn't necessarily reveal the true relationship between variables and outcomes in the real world. Decision-makers are often tempted to view features in SHAP analyses as dials that can be manipulated to engineer specific outcomes, so this distinction must be communicated.