4 ms·

From multi-head to latent attention: The evolution of attention mechanisms

174 points by mgninad 1y ago

mrtesthah 1y ago
Do we know if any of these techniques are actually used in the so-called "frontier" models?
- vinithavn01 1y ago
  The model names are mentioned under each type of attention mechanism
- gchadwick 1y ago
  Who knows what the closed source models use but certainly going by what's happening in open models all the big changes and corresponding gains in capability are in training techniques not model architecture. Things like GQA and MLA as discussed in this article are important techniques for getting better scaling but are relatively minor tweak vs the evolution in training techniques. I suspect closed models aren't doing anything too radically different from what's presented here.
- zackangelo 1y ago
  Not quite a frontier model but definitely built by a frontier lab: Grok 2 was recently open sourced and I believe it uses a fairly standard MHA architecture with MoE.
attogram 1y ago
"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....
- adastra22 1y ago
  Definitely. I always assumed that, having been involved in writing similarly groundbreaking papers… or so we thought at the time. All my coauthors spent significant time thinking about what the best title would be, and strategies like that were common. (It ended up not mattering for us.)
- slickytail 1y ago
  The transformer was a major breakthrough in NLP, and it was clear at the time of publishing that it would have a major impact. But I will add that it is common in the Deep Learning field to give papers catchy titles (see, off the top of my head: all the YOLO papers, ViT, DiT, textual inversion). The transformer paper is one in a long line of seminal papers with funny names.
- sivm 1y ago
  Attention is all you need for what we have. But attention is a local heuristic. We have brittle coherence and no global state. I believe we need a paradigm shift in architecture to move forward.
  - treyd 1y ago
    Has there been research into some hierarchical attention model that has local attention at the scale of sentences and paragraphs that feeds embeddings up to longer range attention across documents?
    - mxkopy 1y ago
      There’s the hierarchical reasoning model https://arxiv.org/abs/2506.21734 https://arxiv.org/abs/2506.21734 but it’s very new and largely untested Though honestly I don’t think new neural network architectures are going to get us over this local maximum, I think the next steps forward involve something that’s 1. Non lossy 2. Readily interpretable
      - miven 1y ago
        The ARC Prize Foundation ran extensive ablations on HRM for their slew of reasoning tasks and noted that the "hierarchical" part of their architecture is not much more impactful than a vanilla transformer of the same size with no extra hyperparameter tuning: https://arcprize.org/blog/hrm-analysis#analyzing-hrms-contribution-to-arc-scores https://arcprize.org/blog/hrm-analysis#analyzing-hrms-contri...
      - ACCount37 1y ago
        By now, I seriously doubt any "readily interpretable" claims. Nothing about human brain is "readily interpretable", and artificial neural networks - which, unlike brains, can be instrumented and experimented on easily - tend to resist interpretation nonetheless. If there was an easy to reduce ML to "readily interpretable" representations, someone would have done so already. If there were architectures that perform similarly but are orders of magnitude more interpretable, they will be used, because interpretability is desirable. Instead, we get what we get.
- iLoveOncall 1y ago
  I recommend reading this article which explains how you can get your papers accepted, and explains that a catchy title is the #1 most important thing: https://maxwellforbes.com/posts/how-to-get-a-paper-accepted/ https://maxwellforbes.com/posts/how-to-get-a-paper-accepted/ (not a plug, I just saved it because it was interesting)
- hyperbovine 1y ago
  It sounds like a typical neurips paper to me. And no, they did know what a big deal it would be, else google never would have given the idea away.
- lucidrains 1y ago
  It is a reference to the beatles song, mainly because Noam Shazeer is a music lover
- ruuda 1y ago
  What about the converse, the paper became some massively influential because of the catchy title? Of course the contents are groundbreaking, but that alone is not enough. A groundbreaking paper that nobody knows about cannot have any impact. Even for research, there is a marketing part to it.
  - eldenring 1y ago
    Huh? of course its enough. Transformers immediately started destroying every single baseline out there. The authors definitely knew it was a very significant discovery beforehand.
    - Mallowram 1y ago
      [dead]
  - soulofmischief 1y ago
    The paper became massively influential because of its contents, not its catchy title. Scientists do not generally read a paper because if its title, they check the abstract and go from there.
- cgearhart 1y ago
  The title is a succinct snippet that spoke directly to researchers at the time. The transformer architecture is somewhat obvious (especially in retrospect), but it was still very surprising because no one was really going this direction. They were going many other directions… That’s the point of the title: you don’t need all kinds of complicated systems for NLP to work—“attention is all you need”. After the success of transfer learning for computer vision in the mid-2010s, it was obvious that NLP needed its own transfer learning approach and AlexNet moment. Lots of research focus around that time was on recurrent models—because that was the conventional wisdom about how you model sequences. Markov chains had led to vanilla RNNs, LSTMs, GRU, etc., which all seemed tantalizingly promising. (MAMBA fans take note.) Attention mechanisms were even used in recurrent models…but so was everything else. Then came transformers—mixing all the then-best-practice bits with the heretical idea of just not giving a shit about O(n^2) complexity. The vanilla transformer used an encoder-decoder structure like the best translation models had been doing; it used a stack of identical blocks to nudge the output along through the pipe like ResNet; it was pretrained on a multi-task objective using a large document corpus. But then it jettisoned all the other complexity and just let it all rest on the attention mechanism to capture long range dependencies. It was immediately thrilling, but it was also completely impractical. (I think the largest model had a 500ish token context limit and bigger than hobbyist GPUs.) So it mostly sat on a shelf while people used other “good enough” models for a few years until the hardware got better and a couple folks proved that it could actually work to run these things at massive scale. And now here we are. I think they knew what they were saying at the time, but I don’t think they knew that it would remain true for years.
  - microtonal 1y ago
    Lots of research focus around that time was on recurrent models—because that was the conventional wisdom about how you model sequences. Markov chains had led to vanilla RNNs, LSTMs, GRU, etc., which all seemed tantalizingly promising. (MAMBA fans take note.) Attention mechanisms were even used in recurrent models…but so was everything else. I feel like there is a step missing here... People were using RNN encoders/decoders for machine translation - the encoder was used to make a representation (fixed-size vector) of the source language sentence, the decoder generated the target language sentence from the source representation. The issue that people were bumping into is that the fixed-sized vector bottlenecked the encoder/decoder architecture. Representing a variable-length source sentence as a fixed-size vector leads to a loss of information that increases with the source sentence length. People started adding attention to the decoder as a way to work around this issue. Each decoder step could attend to every token (well, RNN hidden representation) of the source sentence. So, this led to the RNN + attention architecture. The title 'Attention is all you need' comes from the realization that in this architecture the RNN is not needed, neither for the encoder and decoder. It's a message to the field who was using RNNs + attention (to avoid the bottleneck). Of course, the rest was born from that, encoder-only transformer models like BERT and decoder-only models like current LLMs.
- jjaksic 1y ago
  I was there. They had no idea. The purpose and expectation of the transformer architecture was to enable better translation at scale and without recurrence. This in itself would have been a big deal (and they probably expected some citations), but the actual impact of the architecture was a few orders of magnitude greater.
JSR_FDED 1y ago
Any way to read this without making an account?
- rmonvfer 1y ago
  https://freedium.cfd/https://vinithavn.medium.com/from-multi-head-to-latent-attention-the-evolution-of-attention-mechanisms-64e3c0505f24 https://freedium.cfd/https://vinithavn.medium.com/from-multi...
- kuidaumpf 1y ago
  https://freedium.cfd https://freedium.cfd
- qcnguy 1y ago
  Just click the x at the top right of the interstitial?
  - iLoveOncall 1y ago
    That only work for a few articles per month. But usually opening in incognito does the trick.
- djoldman 1y ago
  just turn off JS.
Mallowram 1y ago
[dead]

174 points by mgninad 1y ago

mrtesthah 1y ago

Do we know if any of these techniques are actually used in the so-called "frontier" models?

vinithavn01 1y ago
The model names are mentioned under each type of attention mechanism
gchadwick 1y ago
Who knows what the closed source models use but certainly going by what's happening in open models all the big changes and corresponding gains in capability are in training techniques not model architecture. Things like GQA and MLA as discussed in this article are important techniques for getting better scaling but are relatively minor tweak vs the evolution in training techniques. I suspect closed models aren't doing anything too radically different from what's presented here.
zackangelo 1y ago
Not quite a frontier model but definitely built by a frontier lab: Grok 2 was recently open sourced and I believe it uses a fairly standard MHA architecture with MoE.

attogram 1y ago

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

adastra22 1y ago
Definitely. I always assumed that, having been involved in writing similarly groundbreaking papers… or so we thought at the time. All my coauthors spent significant time thinking about what the best title would be, and strategies like that were common. (It ended up not mattering for us.)
slickytail 1y ago
The transformer was a major breakthrough in NLP, and it was clear at the time of publishing that it would have a major impact. But I will add that it is common in the Deep Learning field to give papers catchy titles (see, off the top of my head: all the YOLO papers, ViT, DiT, textual inversion). The transformer paper is one in a long line of seminal papers with funny names.
sivm 1y ago
Attention is all you need for what we have. But attention is a local heuristic. We have brittle coherence and no global state. I believe we need a paradigm shift in architecture to move forward.
- treyd 1y ago
  Has there been research into some hierarchical attention model that has local attention at the scale of sentences and paragraphs that feeds embeddings up to longer range attention across documents?
  - mxkopy 1y ago
    There’s the hierarchical reasoning model https://arxiv.org/abs/2506.21734 https://arxiv.org/abs/2506.21734 but it’s very new and largely untested Though honestly I don’t think new neural network architectures are going to get us over this local maximum, I think the next steps forward involve something that’s 1. Non lossy 2. Readily interpretable
    - miven 1y ago
      The ARC Prize Foundation ran extensive ablations on HRM for their slew of reasoning tasks and noted that the "hierarchical" part of their architecture is not much more impactful than a vanilla transformer of the same size with no extra hyperparameter tuning: https://arcprize.org/blog/hrm-analysis#analyzing-hrms-contribution-to-arc-scores https://arcprize.org/blog/hrm-analysis#analyzing-hrms-contri...
    - ACCount37 1y ago
      By now, I seriously doubt any "readily interpretable" claims. Nothing about human brain is "readily interpretable", and artificial neural networks - which, unlike brains, can be instrumented and experimented on easily - tend to resist interpretation nonetheless. If there was an easy to reduce ML to "readily interpretable" representations, someone would have done so already. If there were architectures that perform similarly but are orders of magnitude more interpretable, they will be used, because interpretability is desirable. Instead, we get what we get.
iLoveOncall 1y ago
I recommend reading this article which explains how you can get your papers accepted, and explains that a catchy title is the #1 most important thing: https://maxwellforbes.com/posts/how-to-get-a-paper-accepted/ https://maxwellforbes.com/posts/how-to-get-a-paper-accepted/ (not a plug, I just saved it because it was interesting)
hyperbovine 1y ago
It sounds like a typical neurips paper to me. And no, they did know what a big deal it would be, else google never would have given the idea away.
lucidrains 1y ago
It is a reference to the beatles song, mainly because Noam Shazeer is a music lover
ruuda 1y ago
What about the converse, the paper became some massively influential because of the catchy title? Of course the contents are groundbreaking, but that alone is not enough. A groundbreaking paper that nobody knows about cannot have any impact. Even for research, there is a marketing part to it.
- eldenring 1y ago
  Huh? of course its enough. Transformers immediately started destroying every single baseline out there. The authors definitely knew it was a very significant discovery beforehand.
  - Mallowram 1y ago
    [dead]
- soulofmischief 1y ago
  The paper became massively influential because of its contents, not its catchy title. Scientists do not generally read a paper because if its title, they check the abstract and go from there.
cgearhart 1y ago
The title is a succinct snippet that spoke directly to researchers at the time. The transformer architecture is somewhat obvious (especially in retrospect), but it was still very surprising because no one was really going this direction. They were going many other directions… That’s the point of the title: you don’t need all kinds of complicated systems for NLP to work—“attention is all you need”. After the success of transfer learning for computer vision in the mid-2010s, it was obvious that NLP needed its own transfer learning approach and AlexNet moment. Lots of research focus around that time was on recurrent models—because that was the conventional wisdom about how you model sequences. Markov chains had led to vanilla RNNs, LSTMs, GRU, etc., which all seemed tantalizingly promising. (MAMBA fans take note.) Attention mechanisms were even used in recurrent models…but so was everything else. Then came transformers—mixing all the then-best-practice bits with the heretical idea of just not giving a shit about O(n^2) complexity. The vanilla transformer used an encoder-decoder structure like the best translation models had been doing; it used a stack of identical blocks to nudge the output along through the pipe like ResNet; it was pretrained on a multi-task objective using a large document corpus. But then it jettisoned all the other complexity and just let it all rest on the attention mechanism to capture long range dependencies. It was immediately thrilling, but it was also completely impractical. (I think the largest model had a 500ish token context limit and bigger than hobbyist GPUs.) So it mostly sat on a shelf while people used other “good enough” models for a few years until the hardware got better and a couple folks proved that it could actually work to run these things at massive scale. And now here we are. I think they knew what they were saying at the time, but I don’t think they knew that it would remain true for years.
- microtonal 1y ago
  Lots of research focus around that time was on recurrent models—because that was the conventional wisdom about how you model sequences. Markov chains had led to vanilla RNNs, LSTMs, GRU, etc., which all seemed tantalizingly promising. (MAMBA fans take note.) Attention mechanisms were even used in recurrent models…but so was everything else. I feel like there is a step missing here... People were using RNN encoders/decoders for machine translation - the encoder was used to make a representation (fixed-size vector) of the source language sentence, the decoder generated the target language sentence from the source representation. The issue that people were bumping into is that the fixed-sized vector bottlenecked the encoder/decoder architecture. Representing a variable-length source sentence as a fixed-size vector leads to a loss of information that increases with the source sentence length. People started adding attention to the decoder as a way to work around this issue. Each decoder step could attend to every token (well, RNN hidden representation) of the source sentence. So, this led to the RNN + attention architecture. The title 'Attention is all you need' comes from the realization that in this architecture the RNN is not needed, neither for the encoder and decoder. It's a message to the field who was using RNNs + attention (to avoid the bottleneck). Of course, the rest was born from that, encoder-only transformer models like BERT and decoder-only models like current LLMs.
jjaksic 1y ago
I was there. They had no idea. The purpose and expectation of the transformer architecture was to enable better translation at scale and without recurrence. This in itself would have been a big deal (and they probably expected some citations), but the actual impact of the architecture was a few orders of magnitude greater.

JSR_FDED 1y ago

Any way to read this without making an account?

rmonvfer 1y ago
https://freedium.cfd/https://vinithavn.medium.com/from-multi-head-to-latent-attention-the-evolution-of-attention-mechanisms-64e3c0505f24 https://freedium.cfd/https://vinithavn.medium.com/from-multi...
kuidaumpf 1y ago
https://freedium.cfd https://freedium.cfd
qcnguy 1y ago
Just click the x at the top right of the interstitial?
- iLoveOncall 1y ago
  That only work for a few articles per month. But usually opening in incognito does the trick.
djoldman 1y ago
just turn off JS.

Mallowram 1y ago

[dead]