4 ms·
This is fantastic, I used your earlier transformers article to first get a real grasp on the architecture. I hope you expand this to accommodate other modes of
by Grimm1 6y ago
This is fantastic, I used your earlier transformers article to first get a real grasp on the architecture. I hope you expand this to accommodate other modes of attention outside of transformers paradigm as well!
- jalammar 6y agoWonderful! Thanks! I am curious about those recent O(L) attention transformers (see slide 106 of http://gabrielilharco.com/publications/EMNLP_2020_Tutorial__High_Performance_NLP.pdf http://gabrielilharco.com/publications/EMNLP_2020_Tutorial__...). If these methods are converging towards a new self-attention mechanism, I'd love to try illustrating that. What other attention modes are you referring to? Did something in particular catch your attention?
- Grimm1 6y agoPersonally, I implemented this just yesterday. https://arxiv.org/pdf/1703.03130.pdf https://arxiv.org/pdf/1703.03130.pdf It's a bit older now but I was looking for a self attention method without resorting to a transformer model and this proposed an interesting implementation that wound up being very successful for my problem case.