3 ms·

While I accept your counterargument (esp. regarding credit scores, autonomous vehicles and the like) I think interpretability in deep models is only meaningful

by kavalg 6y ago

While I accept your counterargument (esp. regarding credit scores, autonomous vehicles and the like) I think interpretability in deep models is only meaningful if you have accurate, objective labels. But how often is this the case? For example, when people refer to concepts, they often rely on some established culture, a common understanding, rather than things you can measure. Let's take the simple example of classifying spoiled fruit at the grocery store. You can train a ConvNet and it will probably learn to recognize some visual traits of "spoiledness", but how objective can it really be, given that humans don't always agree what "spoiled" really means? In other words, fruit is spoiled only when a large enough social group says it is spoiled. So if that is your label, then the model can only reflect the shared understanding of "spoiledness" in the given social group. Then interpretability can help you check if the model is looking at plausible things (e.g. the fruit, not the background). However, this won't really tell you "why" the model thinks this fruit is spoiled. You can draw an interesting analogy between this and ensemble models, where the "social group" of the models in the ensemble forms the shared culture. Also and interesting interpretability approach that the article did not mention is SHAP[0] [0] https://github.com/slundberg/shap https://github.com/slundberg/shap

arolihas 6y ago
You could say the same thing about a person’s reasoning. If someone were to think a fruit is spoiled, does that say more about the fruit or about their understanding of spoiledness as you put it?
pkage 6y ago
Regarding accurate labels, a lack of appropriate or sufficient labeling will knock over your model regardless of how powerful your model is (interpretable or not). This is where other benefits of model interpretation come in—you can spot potential errors in your model's training, and that gives you an indication that you need to re-evaluate your base assumptions about the data. SHAP is cool technology! It looks like it builds off LIME and similar to fit a hyperplane against the model surface. I'm not surprised the article didn't mention it though, as it's a bit in the weeds for an overview piece.

by kavalg 6y ago

arolihas 6y ago

You could say the same thing about a person’s reasoning. If someone were to think a fruit is spoiled, does that say more about the fruit or about their understanding of spoiledness as you put it?

pkage 6y ago

Regarding accurate labels, a lack of appropriate or sufficient labeling will knock over your model regardless of how powerful your model is (interpretable or not). This is where other benefits of model interpretation come in—you can spot potential errors in your model's training, and that gives you an indication that you need to re-evaluate your base assumptions about the data. SHAP is cool technology! It looks like it builds off LIME and similar to fit a hyperplane against the model surface. I'm not surprised the article didn't mention it though, as it's a bit in the weeds for an overview piece.