4 ms·

>"What features in the data did the model think are most important?" This is going to be more misleading than informative, since the feature importance hierarc

by nonbel 8y ago

>"What features in the data did the model think are most important?" This is going to be more misleading than informative, since the feature importance hierarchy can change drastically based on what other features are included. > "For any single prediction from a model, how did each feature in the data affect that particular prediction" Ok, but I don't see how this will be comprehended as anything other than a black box for the typical ML model. > "What interactions between features have the biggest effects on a model’s predictions" If there are still big "interaction" effects that means you should have done better feature generation/selection.

iguy 8y ago
> since the feature importance hierarchy can change drastically based on what other features are included This sentence describes a disturbingly large proportion of social science papers. After controlling for X,Y,Z (naturally!) then we find that A is the leading case of B.
- nonbel 8y ago
  Yes, precisely. It is a common practice in medical research as well.
solidasparagus 8y ago
You don't need to explain how the model works, you just need to explain what factors the model cared about when making a certain decision. A strong example is a classifier that distinguishes between wolves and dogs. When you look at which pixels were the most impactful in 'Wolf' predictions, you can see that it is actually pixels of snow that are leading to predictions of 'Wolf'. It's still a bit of a black box, but now it's a black box with an obvious, measurable flaw that you can work to address.
- nonbel 8y ago
  >"A strong example is a classifier that distinguishes between wolves and dogs. When you look at which pixels were the most impactful in 'Wolf' predictions, you can see that it is actually pixels of snow that are leading to predictions of 'Wolf'." Yes, but how common is a situation like this? I mean your classifier is then not much better than making a histogram of pixel intensities.
  - disgruntledphd2 8y ago
    I suspect rather common, but it doesn't get noticed as a lot of people (especially with neural nets), tend not to use feature importance methods, or even examining the residuals.

by nonbel 8y ago

iguy 8y ago

> since the feature importance hierarchy can change drastically based on what other features are included This sentence describes a disturbingly large proportion of social science papers. After controlling for X,Y,Z (naturally!) then we find that A is the leading case of B.

nonbel 8y ago
Yes, precisely. It is a common practice in medical research as well.

solidasparagus 8y ago

You don't need to explain how the model works, you just need to explain what factors the model cared about when making a certain decision. A strong example is a classifier that distinguishes between wolves and dogs. When you look at which pixels were the most impactful in 'Wolf' predictions, you can see that it is actually pixels of snow that are leading to predictions of 'Wolf'. It's still a bit of a black box, but now it's a black box with an obvious, measurable flaw that you can work to address.

nonbel 8y ago
>"A strong example is a classifier that distinguishes between wolves and dogs. When you look at which pixels were the most impactful in 'Wolf' predictions, you can see that it is actually pixels of snow that are leading to predictions of 'Wolf'." Yes, but how common is a situation like this? I mean your classifier is then not much better than making a histogram of pixel intensities.
- disgruntledphd2 8y ago
  I suspect rather common, but it doesn't get noticed as a lot of people (especially with neural nets), tend not to use feature importance methods, or even examining the residuals.