4 ms·

> Intuitively, this seems obvious and valuable — yet when this is mentioned around ML professionals, their faces turn colors and they try to explain that what’s

• ahartman00 9y ago
  Not sure why you were downvoted, I'm assuming this was a good faith question. I'm going to try to answer your question, but I'm still learning, so take this with a grain of salt. Some models have attention, but it is not a requirement. In models that do have attention, it may or may not be simple. In image captioning for example, you can do just that. See [1] for some pictures of this happening. But in this example, they are stepping through a caption, and seeing where the attention is focused for each word. Works fine for a short caption. Videos are just many images and sound, but already this is going to be more difficult. For natural language processing, you will be stepping through very many words. Some models use characters instead of words. Not sure how you could even make sense of that. I have seen people look at the attention in machine translation. Not sure how it would work for sentiment analysis. So you aren't exactly wrong, but you are greatly over simplifying this for the models where it is possible(based on my understanding). Overall, machine learning is very much a black box, despite efforts to the contrary. It also doesn't solve some of the other problems. For example, knowing the model is focusing on zip code doesn't help you remove bias that comes from the data. 1. https://arxiv.org/pdf/1502.03044.pdf https://arxiv.org/pdf/1502.03044.pdf