4 ms·
So far, people have come up with many alternatives for quadratic attention. Only recently have they proven their potential.
by samus 3y ago
So far, people have come up with many alternatives for quadratic attention. Only recently have they proven their potential.
- underlines 3y agotons and tons of papers, most of them had some disadvantages. Can't have the cake and eat it too: https://arxiv.org/html/2404.08801v1 https://arxiv.org/html/2404.08801v1 Meta Megalodon https://arxiv.org/html/2404.07143v1 https://arxiv.org/html/2404.07143v1 Google Infini-Attention https://arxiv.org/html/2402.13753v1 https://arxiv.org/html/2402.13753v1 LongRoPE and a ton more