7 ms·

Understanding and coding the self-attention mechanism of large language models

158 points by mariuz 4y ago

hprotagonist 4y ago
https://arxiv.org/abs/2105.02723 https://arxiv.org/abs/2105.02723 The strong performance of vision transformers on image classification and other vision tasks is often attributed to the design of their multi-head attention layers. However, the extent to which attention is responsible for this strong performance remains unclear. In this short report, we ask: is the attention layer even necessary? Specifically, we replace the attention layer in a vision transformer with a feed-forward layer applied over the patch dimension. The resulting architecture is simply a series of feed-forward layers applied over the patch and feature dimensions in an alternating fashion. In experiments on ImageNet, this architecture performs surprisingly well: a ViT/DeiT-base-sized model obtains 74.9\% top-1 accuracy, compared to 77.9\% and 79.9\% for ViT and DeiT respectively. These results indicate that aspects of vision transformers other than attention, such as the patch embedding, may be more responsible for their strong performance than previously thought. We hope these results prompt the community to spend more time trying to understand why our current models are as effective as they are.
- lostmsu 4y ago
  That's a pretty huge drop in accuracy.
- thomasahle 4y ago
  ViT gives you 90% top-1 accuracy on ImageNet: https://paperswithcode.com/sota/image-classification-on-imagenet https://paperswithcode.com/sota/image-classification-on-imag... I don't know where they get the 77.9% number from. 75% is pretty bad. Similar to the 2015 VGG net as the authors also admit.
  - thomasahle 4y ago
    Nevermind, I guess it's "ImageNet-1K trained models" on which ViT gets 79.9% and the 90% is only when pretraining with ImageNet-22K. There are other non-attention based networks that get 90% too though: https://arxiv.org/pdf/2212.11696v3.pdf https://arxiv.org/pdf/2212.11696v3.pdf
- deleted 4y ago
  [deleted]
mirker 4y ago
Isn’t this very similar to Karpathy’s nanoGPT?
- hummus_bae 4y ago
  nanoGPT is a way to embed low-dimensional embeddings into high-dimensional embeddings, not self-attention.
- taneq 4y ago
  It's talking about large language models and transformers so... yeah? (If you're talking about nanoGPT as a tutorial.)
hackyhacky 4y ago
This explanation walks you through the math and the corresponding code, but (at least in my case, maybe I'm dumb) it failed to help me understand why these steps are necessary or to relate the math to the intended outcome. As a result, I don't feel that I'm any closer to really understanding the heart of self-attention.
- GaggiX 4y ago
  This article explains well why we need attention: https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/ https://jalammar.github.io/visualizing-neural-machine-transl..., and also how it was developed.
  - zmmmmm 4y ago
    those animations are beautiful!
- whoateallthepy 4y ago
  At the end of last year I put together a repository to try and show what is achieved by self-attention on a toy example: detect whether a sequence of characters contains both "a" and "b". The toy problem is useful because the model dimensionality is low enough to make visualization straightforward. The walkthrough also goes through how things can go wrong, and how it can be improved, etc. The walkthrough and code is all available here: https://github.com/rstebbing/workshop/tree/main/experiments/transformer_sequence_classification https://github.com/rstebbing/workshop/tree/main/experiments/.... It's not terse like nanoGPT or similar because the goal is a bit different. In particular, to gain more intuition about the intermediate attention computations, the intermediate tensors are named and persisted so they can be compared and visualized after the fact. Everything should be exactly reproducible locally too!
- doug_durham 4y ago
  I agree. It seems like the target audience is the experienced Deep learning practitioner. Which makes me wonder why such an audience would need this treatment. Why not just read the original paper?
zmmmmm 4y ago
I find all the writeups of self-attention very confusing. They seem to inevitably explain small bits in enormous detail and then leave all kinds of giant areas unexplained, usually are replete with jargon that they don't bother to clarify, etc. I feel like most of these are written by people who are confused themselves, trying to clarify the parts they don't understand as if this is what everybody else must be confused about to, so they inevitably gloss over everything they already know. Fundamentally, I have never understood why a conventional neural network cannot learn self-attention. If layers are fully connected then they definitely have the ability to create a weighting of the inputs accommodating learning of relative spatial features in the data. In fact that's almost a definition of what a neural network is. If relative positional learning is important, that could be added the same way it is in Transformers without the explicit self-attention layer. So what is self-attention really doing beyond this? Why do we need a Query-Key-Value construct here? I am sure I am missing something very basic and fundamental here.
- thisiszilff 4y ago
  If by "conventional neural network" you mean a stack off fully connected layers, then yes, in theory one of those could learn a similar mechanism because of the universal approximation theorem. However, training one might be intractable. It's good to ignore self-attention for a moment and take a look at a convolutional network (a CNN). Why is a CNN more effective than just stacks of fully connected layers? Well, instead of just throwing data at the network and telling it to figure out what to do with it, we've built in some prior knowledge into the network. We tell it "you know, a cup is going to be a cup even if it is 10 pixels up or 10 pixels down; even if it is in the upper right of the image or the lower left." We also tell it, "you know, the pixels near a given pixel are going to be pretty correlated with that pixel, much more so than pixels far away." Convolutions help us express that kind of knowledge in the form of a neural network. Self-attention plays a similar role. We are imbuing our network with an architecture that is aware of the data it is about to receive. We tell it "hey, elements in this sequence have a relation with one another, and that relative location might be important for the answer". Similar to convolutions, we also tell it that the location of various tokens in a sequence is going to vary: there shouldn't be much difference between "Today the dog went to the park" and "The dog went to the park today." Like convolutions, self-attention builds in certain assumptions we have about the data we are going to train the network on. So yes, you are right that fully-connected layers can emulate similar behavior, but training them to do that isn't easy. With self-attention, we've started with more prior knowledge about the problem at hand, so it is easier to solve.
akomtu 4y ago
Is there a coherent theory of ML I could read somewhere? Something higher level than "here is how we use tanh", yet not hand wavy. All these "we've attached this random piece to our model and it makes things better for reasons we don't understand" leaves a poor impression. To an outsider, the GPT model looks a lot like a IIR / FIR filter made of linear operators (matrix multiplications) and convolutions (matrix multiplications in the Fourier space) that processes a sequence of vectors, instead of numbers (like a normal FIR filter would). Perhaps the extensive theory of filters apply here, and perhaps we can even apply the Z transform on the GPT filter to analyze its convergence?
- fooblaster 4y ago
  If I understand correctly z transform analysis can't work as these systems are fundamentally non linear. In fact, the "non linear" part, i.e. the activation functions, are the part of these networks that allow for them to be universal function approximators and learn arbitrary things, versus just becoming a simple linear system.
- PartiallyTyped 4y ago
  I am a big fan of “Geometric Deep Learning” and the view the authors propose. There is also a series of lectures. https://arxiv.org/abs/2104.13478 https://arxiv.org/abs/2104.13478
billconan 4y ago
I can't understand the reason for paying attention to oneself. For example, if we are looking at a seq2seq translation task, does self-attention mean that we "highlight" all words that have similar meanings in a sentence together? What's the intuition that this will help the translation task? for example, in the sentence "I hate apples, I only drink apple juice.", will we encode the first "apples" and "apple" together? why is that useful?
- djsamseng 4y ago
  x = “I hate apples, I only drink _” keys = key_weights * x query = query_weights * x values = value_weights * what_to_look_at For self attention, what_to_look_at = x For regular attention, where_to_look_at could be a database, memory or anything else. So in this example if we’re trying to predict the second “apple” the first “apples” is very helpful. If we’re predicting “juice” then we’d use one head of self-attention to look at the first “apples” and a second head to also look at the second “apple” That’s my understanding at least
- hackinthebochs 4y ago
  >I can't understand the reason for paying attention to oneself. Certain parts of a sentence strongly inform the meaning of other parts and so it is important to encode them together. If you see the word "bank" in a sentence, is it referring to the financial institution or the land next to a body of water? We know by what came before or what comes after. Attention allows relevant context to inform token processing without being distracted by irrelevant context.
- johntb86 4y ago
  In self attention each token creates both "query" and "key" vectors. These are different vectors, so one type of token can look up data for different types of tokens. "Apple" can generated a verb query to look for a verb in the sentence, while also generating a noun key that other words can look for. And with multi-head attention, a single token can get data from different types of tokens. The "self" in self-attention just means that it's looking at other parts of the same string, rather than looking at weights generated from a different string.
typon 4y ago
One thing people generally don't highlight about the transformer architecture, which I think is very important, are the pass through connections. This allows the model to "spread" out it's learnings across the network rather than them being forced to very local. I think this is also a reason why the Resnet was so successful.

158 points by mariuz 4y ago

https://arxiv.org/abs/2105.02723 https://arxiv.org/abs/2105.02723 The strong performance of vision transformers on image classification and other vision tasks is often attributed to the design of their multi-head attention layers. However, the extent to which attention is responsible for this strong performance remains unclear. In this short report, we ask: is the attention layer even necessary? Specifically, we replace the attention layer in a vision transformer with a feed-forward layer applied over the patch dimension. The resulting architecture is simply a series of feed-forward layers applied over the patch and feature dimensions in an alternating fashion. In experiments on ImageNet, this architecture performs surprisingly well: a ViT/DeiT-base-sized model obtains 74.9\% top-1 accuracy, compared to 77.9\% and 79.9\% for ViT and DeiT respectively. These results indicate that aspects of vision transformers other than attention, such as the patch embedding, may be more responsible for their strong performance than previously thought. We hope these results prompt the community to spend more time trying to understand why our current models are as effective as they are.

lostmsu 4y ago
That's a pretty huge drop in accuracy.
thomasahle 4y ago
ViT gives you 90% top-1 accuracy on ImageNet: https://paperswithcode.com/sota/image-classification-on-imagenet https://paperswithcode.com/sota/image-classification-on-imag... I don't know where they get the 77.9% number from. 75% is pretty bad. Similar to the 2015 VGG net as the authors also admit.
- thomasahle 4y ago
  Nevermind, I guess it's "ImageNet-1K trained models" on which ViT gets 79.9% and the 90% is only when pretraining with ImageNet-22K. There are other non-attention based networks that get 90% too though: https://arxiv.org/pdf/2212.11696v3.pdf https://arxiv.org/pdf/2212.11696v3.pdf
deleted 4y ago
[deleted]

mirker 4y ago

Isn’t this very similar to Karpathy’s nanoGPT?

hummus_bae 4y ago
nanoGPT is a way to embed low-dimensional embeddings into high-dimensional embeddings, not self-attention.
taneq 4y ago
It's talking about large language models and transformers so... yeah? (If you're talking about nanoGPT as a tutorial.)

hackyhacky 4y ago

This explanation walks you through the math and the corresponding code, but (at least in my case, maybe I'm dumb) it failed to help me understand why these steps are necessary or to relate the math to the intended outcome. As a result, I don't feel that I'm any closer to really understanding the heart of self-attention.

GaggiX 4y ago
This article explains well why we need attention: https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/ https://jalammar.github.io/visualizing-neural-machine-transl..., and also how it was developed.
- zmmmmm 4y ago
  those animations are beautiful!
whoateallthepy 4y ago
At the end of last year I put together a repository to try and show what is achieved by self-attention on a toy example: detect whether a sequence of characters contains both "a" and "b". The toy problem is useful because the model dimensionality is low enough to make visualization straightforward. The walkthrough also goes through how things can go wrong, and how it can be improved, etc. The walkthrough and code is all available here: https://github.com/rstebbing/workshop/tree/main/experiments/transformer_sequence_classification https://github.com/rstebbing/workshop/tree/main/experiments/.... It's not terse like nanoGPT or similar because the goal is a bit different. In particular, to gain more intuition about the intermediate attention computations, the intermediate tensors are named and persisted so they can be compared and visualized after the fact. Everything should be exactly reproducible locally too!
doug_durham 4y ago
I agree. It seems like the target audience is the experienced Deep learning practitioner. Which makes me wonder why such an audience would need this treatment. Why not just read the original paper?

zmmmmm 4y ago

I find all the writeups of self-attention very confusing. They seem to inevitably explain small bits in enormous detail and then leave all kinds of giant areas unexplained, usually are replete with jargon that they don't bother to clarify, etc. I feel like most of these are written by people who are confused themselves, trying to clarify the parts they don't understand as if this is what everybody else must be confused about to, so they inevitably gloss over everything they already know. Fundamentally, I have never understood why a conventional neural network cannot learn self-attention. If layers are fully connected then they definitely have the ability to create a weighting of the inputs accommodating learning of relative spatial features in the data. In fact that's almost a definition of what a neural network is. If relative positional learning is important, that could be added the same way it is in Transformers without the explicit self-attention layer. So what is self-attention really doing beyond this? Why do we need a Query-Key-Value construct here? I am sure I am missing something very basic and fundamental here.

thisiszilff 4y ago
If by "conventional neural network" you mean a stack off fully connected layers, then yes, in theory one of those could learn a similar mechanism because of the universal approximation theorem. However, training one might be intractable. It's good to ignore self-attention for a moment and take a look at a convolutional network (a CNN). Why is a CNN more effective than just stacks of fully connected layers? Well, instead of just throwing data at the network and telling it to figure out what to do with it, we've built in some prior knowledge into the network. We tell it "you know, a cup is going to be a cup even if it is 10 pixels up or 10 pixels down; even if it is in the upper right of the image or the lower left." We also tell it, "you know, the pixels near a given pixel are going to be pretty correlated with that pixel, much more so than pixels far away." Convolutions help us express that kind of knowledge in the form of a neural network. Self-attention plays a similar role. We are imbuing our network with an architecture that is aware of the data it is about to receive. We tell it "hey, elements in this sequence have a relation with one another, and that relative location might be important for the answer". Similar to convolutions, we also tell it that the location of various tokens in a sequence is going to vary: there shouldn't be much difference between "Today the dog went to the park" and "The dog went to the park today." Like convolutions, self-attention builds in certain assumptions we have about the data we are going to train the network on. So yes, you are right that fully-connected layers can emulate similar behavior, but training them to do that isn't easy. With self-attention, we've started with more prior knowledge about the problem at hand, so it is easier to solve.

akomtu 4y ago

Is there a coherent theory of ML I could read somewhere? Something higher level than "here is how we use tanh", yet not hand wavy. All these "we've attached this random piece to our model and it makes things better for reasons we don't understand" leaves a poor impression. To an outsider, the GPT model looks a lot like a IIR / FIR filter made of linear operators (matrix multiplications) and convolutions (matrix multiplications in the Fourier space) that processes a sequence of vectors, instead of numbers (like a normal FIR filter would). Perhaps the extensive theory of filters apply here, and perhaps we can even apply the Z transform on the GPT filter to analyze its convergence?

fooblaster 4y ago
If I understand correctly z transform analysis can't work as these systems are fundamentally non linear. In fact, the "non linear" part, i.e. the activation functions, are the part of these networks that allow for them to be universal function approximators and learn arbitrary things, versus just becoming a simple linear system.
PartiallyTyped 4y ago
I am a big fan of “Geometric Deep Learning” and the view the authors propose. There is also a series of lectures. https://arxiv.org/abs/2104.13478 https://arxiv.org/abs/2104.13478

billconan 4y ago

I can't understand the reason for paying attention to oneself. For example, if we are looking at a seq2seq translation task, does self-attention mean that we "highlight" all words that have similar meanings in a sentence together? What's the intuition that this will help the translation task? for example, in the sentence "I hate apples, I only drink apple juice.", will we encode the first "apples" and "apple" together? why is that useful?

djsamseng 4y ago
x = “I hate apples, I only drink _” keys = key_weights * x query = query_weights * x values = value_weights * what_to_look_at For self attention, what_to_look_at = x For regular attention, where_to_look_at could be a database, memory or anything else. So in this example if we’re trying to predict the second “apple” the first “apples” is very helpful. If we’re predicting “juice” then we’d use one head of self-attention to look at the first “apples” and a second head to also look at the second “apple” That’s my understanding at least
hackinthebochs 4y ago
>I can't understand the reason for paying attention to oneself. Certain parts of a sentence strongly inform the meaning of other parts and so it is important to encode them together. If you see the word "bank" in a sentence, is it referring to the financial institution or the land next to a body of water? We know by what came before or what comes after. Attention allows relevant context to inform token processing without being distracted by irrelevant context.
johntb86 4y ago
In self attention each token creates both "query" and "key" vectors. These are different vectors, so one type of token can look up data for different types of tokens. "Apple" can generated a verb query to look for a verb in the sentence, while also generating a noun key that other words can look for. And with multi-head attention, a single token can get data from different types of tokens. The "self" in self-attention just means that it's looking at other parts of the same string, rather than looking at weights generated from a different string.

typon 4y ago

One thing people generally don't highlight about the transformer architecture, which I think is very important, are the pass through connections. This allows the model to "spread" out it's learnings across the network rather than them being forced to very local. I think this is also a reason why the Resnet was so successful.