4 ms·
You don't need to explain how the model works, you just need to explain what factors the model cared about when making a certain decision. A strong example is
by solidasparagus 8y ago
You don't need to explain how the model works, you just need to explain what factors the model cared about when making a certain decision.
A strong example is a classifier that distinguishes between wolves and dogs. When you look at which pixels were the most impactful in 'Wolf' predictions, you can see that it is actually pixels of snow that are leading to predictions of 'Wolf'.
It's still a bit of a black box, but now it's a black box with an obvious, measurable flaw that you can work to address.
- nonbel 8y ago>"A strong example is a classifier that distinguishes between wolves and dogs. When you look at which pixels were the most impactful in 'Wolf' predictions, you can see that it is actually pixels of snow that are leading to predictions of 'Wolf'." Yes, but how common is a situation like this? I mean your classifier is then not much better than making a histogram of pixel intensities.
- disgruntledphd2 8y agoI suspect rather common, but it doesn't get noticed as a lot of people (especially with neural nets), tend not to use feature importance methods, or even examining the residuals.