4 ms·
The pair of these papers: (Don't read them in full.) 1.Attention is not explanation (https://arxiv.org/abs/1902.10186 https://arxiv.org/abs/1902.10186) 2.Atte
by screye 6y ago
The pair of these papers: (Don't read them in full.)
1.Attention is not explanation (https://arxiv.org/abs/1902.10186 https://arxiv.org/abs/1902.10186)
2.Attention is not not Explanation (https://arxiv.org/abs/1908.04626 https://arxiv.org/abs/1908.04626)
Goes to show the complete lack of agreement between researchers in the explainability space. Most popular packages (allen NLP, google LIT, Captum) use saliency based methods (Integrated gradients) or Attention. The community has fundamental disagreements on whether they capture anything equivalent to importance as humans would understand it.
An entire community of fairness, ethics and Computational social science is built on top of conclusions using these methods. It is a shame that so much money is poured into these fields, but there does not seem to be as strong a thrust to explore the most fundamental questions themselves.
(my 2 cents: I like SHAP and the stuff coming out of Bin Yu and Tommi Jakkola's labs better..but my opinion too is based in intuition without any real rigor)
- abhgh 6y agoI am following Tommi Jakkola in this space too. Also would recommended Ameet Talwalkar's group, esp the papers with Gregory Plumb like MAPLE [1]. [1] [PDF] https://papers.nips.cc/paper/2018/file/b495ce63ede0f4efc9eec62cb947c162-Paper.pdf https://papers.nips.cc/paper/2018/file/b495ce63ede0f4efc9eec...
- pasiaj 6y agoJaakkola – with two a's. Finnish names are weird.
- monroewalker 6y agoAs a laymen, I don't understand how "attention" and "explanation" are used here. Would you be able to summarize the terms and the contention?
- tmabraham 6y agorecently, a lot of neural network models, especially those in NLP (like GPT-3, BERT, etc.) use "attention" which basically is a way for neural networks to focus on certain subset of the input (the neural network can focus its "attention" to a particular part of the input). Explanations just refers to ways for explaining the predictions of the neural networks.
- zamfi 6y agoThese are deep neural net papers, specifically in NLP. Explanation: why the model (the deep neural net, that is) is doing what it's doing. Attention: a particular technique used in certain deep net models, invented a few years ago, that originally showed remarkable performance improvements in NLP (natural language processing) tasks, but has recently been applied in vision and elsewhere.
- charlescearl 6y agoSome more links provided at a NeurIPS tutorial this week https://explainml-tutorial.github.io/papers https://explainml-tutorial.github.io/papers I saw the first paper mentioned in the tutorial chat. What is your impression of anchor? https://github.com/marcotcr/anchor https://github.com/marcotcr/anchor