3 ms·
Can you talk about the attention visualization a bit. What’s the benefit. I have been plotting attention masks and even visualizing them over sentences but it o
by 2099miles 3y ago
Can you talk about the attention visualization a bit. What’s the benefit. I have been plotting attention masks and even visualizing them over sentences but it obv doesn’t result in a ton of insight due to the sentences changing from words to much more complex vectors that represent those words after being processed through attentions 32x.
- malux85 3y agoI am using attention weighted pooling strategy to generate cross-attentions across N transformer heads, this allows us to visualize which parts of an output were caused by which part of the input -- this is used to debug summarization. Only works for encoder/decoder networks - i.e. ones where you have cross attentions