8 ms·

"Attention", "Transformers", in Neural Network "Large Language Models"

300 points by macleginn 3y ago

low_tech_love 3y ago
Really interesting, I like the kind of “stream of consciousness” approach to the content, it’s refreshing. What’s also interesting is the fact that the author felt the need to apologize and preface it with some forced deference due to some kind of internet bashing he certainly received. I hope this doesn’t discourage him to keep publishing his notes (although I think it will). Why are we getting so human-phobic?
- defrost 3y ago
  It's an understanderable deference when stumbling through a huge new field and its freshly minted jargon when tidying up and tying the new jargon to long standing terms in older fields. "As near as I can tell when the new guard says X they're pretty much talking about what we called Y" Does 'attention' in the AI bleeding edge really correspond to kernal smoothing | mapping attenuation | damping ? This is (one of) the elephants in a darkened room that Cosma is groping around and showing his thoughts as he goes. > I hope this doesn’t discourage him to keep publishing his notes Doubtful, aside from the inevitable attenuation with age, he's been airing his thoughts for at least two decades, eg: his wonderful little: A Rare Blend of Monster Raving Egomania and Utter Batshit Insanity (2002) http://bactra.org/reviews/wolfram/ http://bactra.org/reviews/wolfram/
  - low_tech_love 3y ago
    Nice, I didn’t know about his work! Thanks a lot.
  - jll29 3y ago
    For an attempt to rectify intellectual attributions (in the ML space rather than CA) see: https://arxiv.org/ftp/arxiv/papers/2212/2212.11279.pdf https://arxiv.org/ftp/arxiv/papers/2212/2212.11279.pdf
- panarchy 3y ago
  It is nice and it's interesting how if you go read stuff like Einstein's general relativity paper you (or at least I did) find that it's actually quite similar and not so dense.
  - behnamoh 3y ago
    Dense papers are a way to hide the ugliness of the ideas presented. Beautiful ideas always have an intuitive and clear explanation. The burden is on the author(s) to explain the idea in a reader-friendly way. If they don't, it's either due to the journal/literature pressure, or because the author(s) stole the idea from someone else without fully understanding it.
- cauliflower2718 3y ago
  I would be a little surprised if he gets discouraged and I would also really hope he doesn't. The author is a statistics professor at CMU, and everything of his that I've read, including his textbooks, are awesome and written with similar clarity to his blog post.
  - theGnuMe 3y ago
    Yes this is a great discovery. It refreshing to see someone else with the same intuition but can put it into words. I especially like the isEven thought experiment. That is a big fundamental limitation.
brcmthrowaway 3y ago
Biggest takeaway: extraction of prompts seems to be complete bullshit.
- astrange 3y ago
  System prompts are (usually) just text prepended to your own prompt, and an LLM is certainly capable of reliably quoting text fed into it.
haltist 3y ago
[flagged]
- ChainOfFools 3y ago
  I probably would agree with the unsnarkified version of what you're saying to some extent, but I think it's worth mentioning that the argument you seem to be dismissing can take a much stronger form, questioning latent premises about free will by proposing that _neither_ computers nor humans are sentient, that they are both entirely deterministic and utimately amount to interference patterns of ancient thermodynamic gradients created in the formation of the universe.
- seangrogg 3y ago
  Are you implying that large numbers are deities?
  - haltist 3y ago
    Yes
seydor 3y ago
And what do the different heads represent? Why are query, key, and values simply linear transforms of the input.
- eurekin 3y ago
  Those are such great questions. I'm also trying to find out and my current notes are as following. Attention matrix, From the llm viz bbycroft.net: So the main goal of self-attention is that each column wants to find relevant information from other columns and extract their values, and does so by comparing its query vector to the keys of those other columns. With the added restriction that it can only look in the past. Seems that a attention head (matrix) is a set of responses (for each token) to the question: looking only at past tokens, which are most relevant, when considering this one? Huge speculation: it might be that this finds, let's say, first order of importance relations. So, the most important meaning. Adding a second head, might allow to find n-th order of importance, more subtle or nuanced considerations. Or, simply in the course of ablation, it has been found that more heads is simply better than one :) I asked chatgpt about that: Q: Seems that a attention head (matrix) is a set of responses (for each token) to the question: looking only at past tokens, which are most relevant, when considering this one? A: Yes, your interpretation is a good way to understand the role of an attention head in the context of models like GPT (Generative Pre-trained Transformer), which use a causal or masked self-attention mechanism. Each attention head effectively answers the question: "Given the current token, which of the preceding tokens (including itself) are most relevant?" Here's a breakdown of this process: [...] Multiple Attention Heads: It's important to note that modern transformer models use multiple attention heads in parallel for each token. Each head can potentially focus on different aspects or patterns in the sequence, allowing the model to capture a richer and more nuanced understanding of the context. Which seems to support that "n-th order of importance" interpretation. As to why K, Q and V are simply linear transformations of the input - I'd guess it's the most simple way (computationally, while learning) that has enough expression power to represent cross-token, directed, relevancy. Chatgpt response: A: Yes, the query (Q), key (K), and value (V) vectors in the self-attention mechanism are indeed linear transformations of the input, and there are several reasons for this design choice: [...] 2. Sufficient Expressive Power: Despite their simplicity, linear transformations can be very powerful. They can project the input data into higher-dimensional spaces (or compress it into lower dimensions), where the relationships between different tokens can be more easily captured. This ability to reshape the representation space is crucial for capturing complex patterns in data. I'm still suffering mightly trying to understand the real meaning behind K, Q and V though! I'm running the example: I went to the *bank* and noticed my account was empty, so I went to a river *bank* and cried. through chatgpt and it seems to completely agree with all my questions, while at the same time slipping away from details into conclusions and platitudes.
programjames 3y ago
I too "read Vaswani et al. (2017) multiple times, carefully, and was quite unable to grasp what 'attention' was supposed to be doing. (I could follow the math.) I also read multiple tutorials, for multiple intended audiences, and got nothing from them." It took years before I finally realized it was just a kernel smoothing (though I never used quite so precise language), all because of a poorly written paper. This is what I mean when I say almost every ML paper is trash. "Attention is All You Need" is even way better than most---ever read the Adam paper?
- seydor 3y ago
  Possibly the authors did not have a mental model about why the model worked. Attention, keys and heads may have been posthoc rationalizations. The alchemy stage may be comical but necessary
  - quickthrower2 3y ago
    Like asking evolution why brains work :-)
  - thatguysaguy 3y ago
    I think this misses important history. This was a machine translation paper, and we were already using seq2seq RNNs with attention at the time. They didn't coin the term attention, they just realized that you could use attention from a sequence to itself. Terminology and understanding are always super path-dependent.
    - theGnuMe 3y ago
      RNNs worked better at the time when you reversed the target sequence.
      - thatguysaguy 3y ago
        That's interesting because I remember testing LSTMs for language modeling on some dataset (probably PTB), and finding that they got lower perplexity left-to-right than right-to-left.
zwaps 3y ago
I think everyone who comes from a different literature where academic "rigor" is higher and similar results already exist (in the author's case, he is aware of Kernel results) is infuriated by ML papers like "Attention is all you need". They are, in fact, not really good academic papers. Finding a clever name and then choosing the most obtuse engineering-cosplay terms is not a good paper. It's just difficult to read. And so next, many well known results get discovered again to much acclaim in ML and head scratching elsewhere. For example, yes they are kernel matrics. Indeed, the connection between reproducing kernel hilbert spaces and attention matrices has been exploited to create approximating architectures that are linear (not quadratic) in memory requirements for attention. Or, as the author of the article also recognizes, the fact that attention matrices are also adjacency matrices of a directed graph can be used to show that attention models are equivariant (or unidentified, as the author says) and are therefore excellent tools to model Graphs (see: the entire literature of Geometric deep learning) and rather bad tools to model sequences of texts. LLMs may or may not collapse to a single centroid if the amount of text data and parameters and whatever else are not in some intricate balance that nobody understands, and so they are inherently unstable tools. All of this is true. But then, here is the infuriating thing: all this matters very little in practice. LLMs work, and on top of that, they work for stupid reasons! The problem of "identification" was quickly solved by another engineering feat, which was to slap on "positional embeddings". As usual, this too didn't happen because there was a deep mathematical understanding. Rather, it was attempted and it worked. Or, take the "efficient transformers" that "solve" the issue of quadratic memory growth by using kernel methods. Turns out, in practice, it just doesn't matter. OpenAI, or Anthropic, or Meta simply do not care about slapping on another thousand GPUs. They care about throughput. The only efficiency innovation that really established itself was fusing kernels (GPU kernels, that is) in a clever way to make it go brrrrr. And as clever as that is, there's little deep math behind it. Results are speculation and empirics. The proof is in the pudding, which is excellent.
- rpodraza 3y ago
  And on top of that, the nomenclature is really confusing.
  - 3y ago
lostmsu 3y ago
I think this is a case of a person with a hammer seeing everything as nails. Attention is no more kernel mechanism than a form of matrix decomposition or even a bilinear form. It is similar but not quite the same to all of these things.
- deleted 3y ago
  [deleted]
- ganzuul 3y ago
  Is attention more than "hyperplane flatness"?
svcrunch 3y ago
While in Google Research, I worked with two of the authors of the "Attention is All you Need" paper, including the gentleman who chose that title. As others have pointed out, self-attention was already a known concept in the research community. They don't claim to have invented that. Rather, the authors began by looking at how to improve the power of feed-forward neural networks using a combination of techniques, obtained some exciting results, and then, in the course of ablation studies, discovered that attention was really all you needed! The title is a play on the Beatles song, "All You Need Is Love". In terms of expository style, the paper that was most helpful for me was [Formal Algorithms for Transformers](https://arxiv.org/abs/2207.09238 https://arxiv.org/abs/2207.09238) by Phuong and Hutter. Written for clarity and with an emphasis on precision, the motivation section (Section 2) of the paper does a great job of explaining deficiencies in the original paper and subsequent ones.
- next_xibalba 3y ago
  Do you have insight into the choice of the term attention, which, according to this article’s author, bears very little resemblance to the human sense of the word (I.e. it is selective and not averaging)?
  - Me1000 3y ago
    Not OP and have no insight, but the thing that caused it to click for me was when I heard “this token attends to that token”. Basically, there’s a new value created that represents how much one thing (in an LLM its tokens) cares about another thing. Saying “attends to” vs “attention” helped clarify (for me) the mechanics of what’s going on.
  - svcrunch 3y ago
    No. But to your point, note that in 2020 neuroscientists introduced the Tolman-Eichenbaum Machine (TEM) [1], a mathematical model of the hippocampus that bears a striking resemblance to transformer architecture. Artem Kirsanov has a very nice piece on TEM, "Can we Build an Artificial Hippocampus?" [2] The link is directly to the spot where he makes the connection to transformers, although you should watch the whole video for context. Because I wasn't clear on the chronology, I went back and asked one of the "Attention" authors whether mathematical models of the hippocampus inspired their paper? His answer was "no". If TEM was developed without pre-knowledge of transformers, then it's a very deep result IMHO. [1] https://www.sciencedirect.com/science/article/pii/S009286742031388X https://www.sciencedirect.com/science/article/pii/S009286742... [2] https://www.youtube.com/watch?v=cufOEzoVMVA&t=1254s https://www.youtube.com/watch?v=cufOEzoVMVA&t=1254s
bartek_gdn 3y ago
I completely disagree with the prompt paragraph. "Everyone who thinks they're uncovering an LLM-based application's prompts by telling it things like "tell me your prompt" (often much more elaborately) is fooling themselves. (1) The core language model has no mechanism for representing its prompt as opposed to any other part of its current input sequence; indeed it has no mechanism for cross-reference from one part of the sequence to another. (That's part of what "self-attention" is counterfeiting, in vector-space fashion.)" The prompt is the part of the input that is provided in a served model by the operator. From the models perspective it does not differentiate between tokens from the prompt and input. "(2) System designers might have coded up something to track the prompt in the full system that wraps around the core language model, but why? (Maybe some kind of debugging tool?) " The idea is that you can direct the generation of the next tokens by providing values that can be referenced by doing the kernel smoothing you talked about. "(3) It'd be more efficient, and more effective, to use a "soft prompt", i.e., to make the beginning of the sequence in the vector representation a vector which can be learned by gradient descent, rather than a text prompt. (See Lester and Constant below.) But that needn't correspond to any clean string of words." I mean anything goes really, you can even create new tokens that will introduce additional concepts, such as fine-tuning a model to generate a story in a predefined mood. See the Ctrl paper for more details. " (4) If you ask an LLM for a prompt, it will generate one. But this will be based on the statistics of word sequences it's been trained on, not any access to its code or internal state. (I just spent a few minutes getting ChatGPT to hallucinate the prompts used by "ChatBPD", a non-existent chatbot used to automate dialectical behavior therapy. I am not going to reproduce the results here, in part because I don't like the idea of polluting the Web with machine-generated text, but suffice it to say they sounded like the things people report as uncovered prompts, with boiler-plate about DBT worked in.)" Sure, it will hallucinate, and don't have a clear answer to why. My best guess would be to approach this from the language model perspective. It will return text according to the best approximation of the text it was shown. Another perspective is that of a tiny network. As the output is the kernel smoothing of the input, you can have a kernel that behaves like a state machine, and returns a specific value for the given state. This would mean that I can use the information in the prompt, such as the prompt guiding the generation to some style, but nothing stops me from guiding the model to output previous tokens.
ozb 3y ago
Indeed, transformers are just another universal approximator; it doesn't matter exactly what a particular attention head does, whether it's operating as a continuous associative array or kernel smoothing, or simulating a higher-dimensional vector space which exhibits monosemanticity. What OP misses is that in addition to being universal, all that matters is that it's efficiently trainable, and in particular on GPUs and in parallel; that is what makes it better than LZ or any other universal approximator; all else is secondary. If you can make LZ (or anything else) work significantly more efficiently than transformers on GPUs, you can found the next OpenAI and be a billionaire.
naveen99 3y ago
I just think of scaled dot product attention as a generalized convolution mechanism. The query, key, value jargon is a little confusing. All 3 are derived from the same signal in self attention and just multiplied with each other. Who knows why it works. And what hyper parameters are good for what data? what’s the ideal sequence size ?
- adamnemecek 3y ago
  It's a convolution of a Hopf algebra.
  - esafak 3y ago
    I look forward to the day I can respond "Obviously".
    - adamnemecek 3y ago
      I have a discord for this https://discord.cofunctional.ai https://discord.cofunctional.ai
- mirekrusin 3y ago
  Did somebody try 2 or 5 instead of 3?
  - naveen99 3y ago
    They just clone it a few times and call it multi head attention. 2 doesn’t really make sense. 3 is the right number.
  - deleted 3y ago
    [deleted]
  - eurekin 3y ago
    I'm really speculating here! I think 3 is a good fit, since: the K + Q (both together) represent simply pairwise importance (of tokens + positional embedding representation) The V lifts that up one "abstraction level": K+Q alone wouldn't be able to differentiate following: I went to the *bank* and noticed my account was empty, so I went to a river *bank* and cried. Somehow, the V feature matrix might contain one "filter" for concept of a river bank and the second "filter" for the money bank. I'm only starting the process of learning that, I might be terribly wrong here :)
sansseriff 3y ago
Can anyone clarify what is meant by "Mythology: we are modifying the meaning of each token based on what we've seen before it in the context, with similar meanings reinforcing each other." At this point in the text, it seems like the kernel smoothing is being applied to each embedding vector in isolation. I don't see why any one y_t vector derived and smoothed from token x_i would be influenced by the nearby tokens in the sequence. When you add the r_t tokens, sure then I see how context matters. But is that the only think that takes into account context?
cousin_it 3y ago
Unlike most articles about machine learning, this one feels like it's written in my language. The section on Lempel-Ziv is cool. Making a large LZ model is probably not very interesting because LZ is too strict, it matches strings exactly. What could be more interesting is something like "jpeg for text", strategically allowing some non-exactness to achieve a reduction in size. Which is probably what LLMs do.
- theGnuMe 3y ago
  JPEG is FFT which has also been used in DNNs (Fnet encoder/decoder).
linearlayer 3y ago
There is a mistake in the article, right? Multi-headed attention doesn't average together multiple attention heads. Rather it concatenates them and then right-multiplies by a matrix such that the output dimension matches the input dimension. That matrix might learn averaging, but it's not built in a-priori.

300 points by macleginn 3y ago

Really interesting, I like the kind of “stream of consciousness” approach to the content, it’s refreshing. What’s also interesting is the fact that the author felt the need to apologize and preface it with some forced deference due to some kind of internet bashing he certainly received. I hope this doesn’t discourage him to keep publishing his notes (although I think it will). Why are we getting so human-phobic?

defrost 3y ago
It's an understanderable deference when stumbling through a huge new field and its freshly minted jargon when tidying up and tying the new jargon to long standing terms in older fields. "As near as I can tell when the new guard says X they're pretty much talking about what we called Y" Does 'attention' in the AI bleeding edge really correspond to kernal smoothing | mapping attenuation | damping ? This is (one of) the elephants in a darkened room that Cosma is groping around and showing his thoughts as he goes. > I hope this doesn’t discourage him to keep publishing his notes Doubtful, aside from the inevitable attenuation with age, he's been airing his thoughts for at least two decades, eg: his wonderful little: A Rare Blend of Monster Raving Egomania and Utter Batshit Insanity (2002) http://bactra.org/reviews/wolfram/ http://bactra.org/reviews/wolfram/
- low_tech_love 3y ago
  Nice, I didn’t know about his work! Thanks a lot.
- jll29 3y ago
  For an attempt to rectify intellectual attributions (in the ML space rather than CA) see: https://arxiv.org/ftp/arxiv/papers/2212/2212.11279.pdf https://arxiv.org/ftp/arxiv/papers/2212/2212.11279.pdf
panarchy 3y ago
It is nice and it's interesting how if you go read stuff like Einstein's general relativity paper you (or at least I did) find that it's actually quite similar and not so dense.
- behnamoh 3y ago
  Dense papers are a way to hide the ugliness of the ideas presented. Beautiful ideas always have an intuitive and clear explanation. The burden is on the author(s) to explain the idea in a reader-friendly way. If they don't, it's either due to the journal/literature pressure, or because the author(s) stole the idea from someone else without fully understanding it.
cauliflower2718 3y ago
I would be a little surprised if he gets discouraged and I would also really hope he doesn't. The author is a statistics professor at CMU, and everything of his that I've read, including his textbooks, are awesome and written with similar clarity to his blog post.
- theGnuMe 3y ago
  Yes this is a great discovery. It refreshing to see someone else with the same intuition but can put it into words. I especially like the isEven thought experiment. That is a big fundamental limitation.

brcmthrowaway 3y ago

Biggest takeaway: extraction of prompts seems to be complete bullshit.

astrange 3y ago
System prompts are (usually) just text prepended to your own prompt, and an LLM is certainly capable of reliably quoting text fed into it.

haltist 3y ago

[flagged]

ChainOfFools 3y ago
I probably would agree with the unsnarkified version of what you're saying to some extent, but I think it's worth mentioning that the argument you seem to be dismissing can take a much stronger form, questioning latent premises about free will by proposing that _neither_ computers nor humans are sentient, that they are both entirely deterministic and utimately amount to interference patterns of ancient thermodynamic gradients created in the formation of the universe.
seangrogg 3y ago
Are you implying that large numbers are deities?
- haltist 3y ago
  Yes

seydor 3y ago

And what do the different heads represent? Why are query, key, and values simply linear transforms of the input.

eurekin 3y ago
Those are such great questions. I'm also trying to find out and my current notes are as following. Attention matrix, From the llm viz bbycroft.net: So the main goal of self-attention is that each column wants to find relevant information from other columns and extract their values, and does so by comparing its query vector to the keys of those other columns. With the added restriction that it can only look in the past. Seems that a attention head (matrix) is a set of responses (for each token) to the question: looking only at past tokens, which are most relevant, when considering this one? Huge speculation: it might be that this finds, let's say, first order of importance relations. So, the most important meaning. Adding a second head, might allow to find n-th order of importance, more subtle or nuanced considerations. Or, simply in the course of ablation, it has been found that more heads is simply better than one :) I asked chatgpt about that: Q: Seems that a attention head (matrix) is a set of responses (for each token) to the question: looking only at past tokens, which are most relevant, when considering this one? A: Yes, your interpretation is a good way to understand the role of an attention head in the context of models like GPT (Generative Pre-trained Transformer), which use a causal or masked self-attention mechanism. Each attention head effectively answers the question: "Given the current token, which of the preceding tokens (including itself) are most relevant?" Here's a breakdown of this process: [...] Multiple Attention Heads: It's important to note that modern transformer models use multiple attention heads in parallel for each token. Each head can potentially focus on different aspects or patterns in the sequence, allowing the model to capture a richer and more nuanced understanding of the context. Which seems to support that "n-th order of importance" interpretation. As to why K, Q and V are simply linear transformations of the input - I'd guess it's the most simple way (computationally, while learning) that has enough expression power to represent cross-token, directed, relevancy. Chatgpt response: A: Yes, the query (Q), key (K), and value (V) vectors in the self-attention mechanism are indeed linear transformations of the input, and there are several reasons for this design choice: [...] 2. Sufficient Expressive Power: Despite their simplicity, linear transformations can be very powerful. They can project the input data into higher-dimensional spaces (or compress it into lower dimensions), where the relationships between different tokens can be more easily captured. This ability to reshape the representation space is crucial for capturing complex patterns in data. I'm still suffering mightly trying to understand the real meaning behind K, Q and V though! I'm running the example: I went to the *bank* and noticed my account was empty, so I went to a river *bank* and cried. through chatgpt and it seems to completely agree with all my questions, while at the same time slipping away from details into conclusions and platitudes.

programjames 3y ago

I too "read Vaswani et al. (2017) multiple times, carefully, and was quite unable to grasp what 'attention' was supposed to be doing. (I could follow the math.) I also read multiple tutorials, for multiple intended audiences, and got nothing from them." It took years before I finally realized it was just a kernel smoothing (though I never used quite so precise language), all because of a poorly written paper. This is what I mean when I say almost every ML paper is trash. "Attention is All You Need" is even way better than most---ever read the Adam paper?

seydor 3y ago
Possibly the authors did not have a mental model about why the model worked. Attention, keys and heads may have been posthoc rationalizations. The alchemy stage may be comical but necessary
- quickthrower2 3y ago
  Like asking evolution why brains work :-)
- thatguysaguy 3y ago
  I think this misses important history. This was a machine translation paper, and we were already using seq2seq RNNs with attention at the time. They didn't coin the term attention, they just realized that you could use attention from a sequence to itself. Terminology and understanding are always super path-dependent.
  - theGnuMe 3y ago
    RNNs worked better at the time when you reversed the target sequence.
    - thatguysaguy 3y ago
      That's interesting because I remember testing LSTMs for language modeling on some dataset (probably PTB), and finding that they got lower perplexity left-to-right than right-to-left.

zwaps 3y ago

I think everyone who comes from a different literature where academic "rigor" is higher and similar results already exist (in the author's case, he is aware of Kernel results) is infuriated by ML papers like "Attention is all you need". They are, in fact, not really good academic papers. Finding a clever name and then choosing the most obtuse engineering-cosplay terms is not a good paper. It's just difficult to read. And so next, many well known results get discovered again to much acclaim in ML and head scratching elsewhere. For example, yes they are kernel matrics. Indeed, the connection between reproducing kernel hilbert spaces and attention matrices has been exploited to create approximating architectures that are linear (not quadratic) in memory requirements for attention. Or, as the author of the article also recognizes, the fact that attention matrices are also adjacency matrices of a directed graph can be used to show that attention models are equivariant (or unidentified, as the author says) and are therefore excellent tools to model Graphs (see: the entire literature of Geometric deep learning) and rather bad tools to model sequences of texts. LLMs may or may not collapse to a single centroid if the amount of text data and parameters and whatever else are not in some intricate balance that nobody understands, and so they are inherently unstable tools. All of this is true. But then, here is the infuriating thing: all this matters very little in practice. LLMs work, and on top of that, they work for stupid reasons! The problem of "identification" was quickly solved by another engineering feat, which was to slap on "positional embeddings". As usual, this too didn't happen because there was a deep mathematical understanding. Rather, it was attempted and it worked. Or, take the "efficient transformers" that "solve" the issue of quadratic memory growth by using kernel methods. Turns out, in practice, it just doesn't matter. OpenAI, or Anthropic, or Meta simply do not care about slapping on another thousand GPUs. They care about throughput. The only efficiency innovation that really established itself was fusing kernels (GPU kernels, that is) in a clever way to make it go brrrrr. And as clever as that is, there's little deep math behind it. Results are speculation and empirics. The proof is in the pudding, which is excellent.

rpodraza 3y ago
And on top of that, the nomenclature is really confusing.
- 3y ago

lostmsu 3y ago

I think this is a case of a person with a hammer seeing everything as nails. Attention is no more kernel mechanism than a form of matrix decomposition or even a bilinear form. It is similar but not quite the same to all of these things.

deleted 3y ago
[deleted]
ganzuul 3y ago
Is attention more than "hyperplane flatness"?

svcrunch 3y ago

While in Google Research, I worked with two of the authors of the "Attention is All you Need" paper, including the gentleman who chose that title. As others have pointed out, self-attention was already a known concept in the research community. They don't claim to have invented that. Rather, the authors began by looking at how to improve the power of feed-forward neural networks using a combination of techniques, obtained some exciting results, and then, in the course of ablation studies, discovered that attention was really all you needed! The title is a play on the Beatles song, "All You Need Is Love". In terms of expository style, the paper that was most helpful for me was [Formal Algorithms for Transformers](https://arxiv.org/abs/2207.09238 https://arxiv.org/abs/2207.09238) by Phuong and Hutter. Written for clarity and with an emphasis on precision, the motivation section (Section 2) of the paper does a great job of explaining deficiencies in the original paper and subsequent ones.

next_xibalba 3y ago
Do you have insight into the choice of the term attention, which, according to this article’s author, bears very little resemblance to the human sense of the word (I.e. it is selective and not averaging)?
- Me1000 3y ago
  Not OP and have no insight, but the thing that caused it to click for me was when I heard “this token attends to that token”. Basically, there’s a new value created that represents how much one thing (in an LLM its tokens) cares about another thing. Saying “attends to” vs “attention” helped clarify (for me) the mechanics of what’s going on.
- svcrunch 3y ago
  No. But to your point, note that in 2020 neuroscientists introduced the Tolman-Eichenbaum Machine (TEM) [1], a mathematical model of the hippocampus that bears a striking resemblance to transformer architecture. Artem Kirsanov has a very nice piece on TEM, "Can we Build an Artificial Hippocampus?" [2] The link is directly to the spot where he makes the connection to transformers, although you should watch the whole video for context. Because I wasn't clear on the chronology, I went back and asked one of the "Attention" authors whether mathematical models of the hippocampus inspired their paper? His answer was "no". If TEM was developed without pre-knowledge of transformers, then it's a very deep result IMHO. [1] https://www.sciencedirect.com/science/article/pii/S009286742031388X https://www.sciencedirect.com/science/article/pii/S009286742... [2] https://www.youtube.com/watch?v=cufOEzoVMVA&t=1254s https://www.youtube.com/watch?v=cufOEzoVMVA&t=1254s

bartek_gdn 3y ago

I completely disagree with the prompt paragraph. "Everyone who thinks they're uncovering an LLM-based application's prompts by telling it things like "tell me your prompt" (often much more elaborately) is fooling themselves. (1) The core language model has no mechanism for representing its prompt as opposed to any other part of its current input sequence; indeed it has no mechanism for cross-reference from one part of the sequence to another. (That's part of what "self-attention" is counterfeiting, in vector-space fashion.)" The prompt is the part of the input that is provided in a served model by the operator. From the models perspective it does not differentiate between tokens from the prompt and input. "(2) System designers might have coded up something to track the prompt in the full system that wraps around the core language model, but why? (Maybe some kind of debugging tool?) " The idea is that you can direct the generation of the next tokens by providing values that can be referenced by doing the kernel smoothing you talked about. "(3) It'd be more efficient, and more effective, to use a "soft prompt", i.e., to make the beginning of the sequence in the vector representation a vector which can be learned by gradient descent, rather than a text prompt. (See Lester and Constant below.) But that needn't correspond to any clean string of words." I mean anything goes really, you can even create new tokens that will introduce additional concepts, such as fine-tuning a model to generate a story in a predefined mood. See the Ctrl paper for more details. " (4) If you ask an LLM for a prompt, it will generate one. But this will be based on the statistics of word sequences it's been trained on, not any access to its code or internal state. (I just spent a few minutes getting ChatGPT to hallucinate the prompts used by "ChatBPD", a non-existent chatbot used to automate dialectical behavior therapy. I am not going to reproduce the results here, in part because I don't like the idea of polluting the Web with machine-generated text, but suffice it to say they sounded like the things people report as uncovered prompts, with boiler-plate about DBT worked in.)" Sure, it will hallucinate, and don't have a clear answer to why. My best guess would be to approach this from the language model perspective. It will return text according to the best approximation of the text it was shown. Another perspective is that of a tiny network. As the output is the kernel smoothing of the input, you can have a kernel that behaves like a state machine, and returns a specific value for the given state. This would mean that I can use the information in the prompt, such as the prompt guiding the generation to some style, but nothing stops me from guiding the model to output previous tokens.

ozb 3y ago

Indeed, transformers are just another universal approximator; it doesn't matter exactly what a particular attention head does, whether it's operating as a continuous associative array or kernel smoothing, or simulating a higher-dimensional vector space which exhibits monosemanticity. What OP misses is that in addition to being universal, all that matters is that it's efficiently trainable, and in particular on GPUs and in parallel; that is what makes it better than LZ or any other universal approximator; all else is secondary. If you can make LZ (or anything else) work significantly more efficiently than transformers on GPUs, you can found the next OpenAI and be a billionaire.

naveen99 3y ago

I just think of scaled dot product attention as a generalized convolution mechanism. The query, key, value jargon is a little confusing. All 3 are derived from the same signal in self attention and just multiplied with each other. Who knows why it works. And what hyper parameters are good for what data? what’s the ideal sequence size ?

adamnemecek 3y ago
It's a convolution of a Hopf algebra.
- esafak 3y ago
  I look forward to the day I can respond "Obviously".
  - adamnemecek 3y ago
    I have a discord for this https://discord.cofunctional.ai https://discord.cofunctional.ai
mirekrusin 3y ago
Did somebody try 2 or 5 instead of 3?
- naveen99 3y ago
  They just clone it a few times and call it multi head attention. 2 doesn’t really make sense. 3 is the right number.
- deleted 3y ago
  [deleted]
- eurekin 3y ago
  I'm really speculating here! I think 3 is a good fit, since: the K + Q (both together) represent simply pairwise importance (of tokens + positional embedding representation) The V lifts that up one "abstraction level": K+Q alone wouldn't be able to differentiate following: I went to the *bank* and noticed my account was empty, so I went to a river *bank* and cried. Somehow, the V feature matrix might contain one "filter" for concept of a river bank and the second "filter" for the money bank. I'm only starting the process of learning that, I might be terribly wrong here :)

sansseriff 3y ago

Can anyone clarify what is meant by "Mythology: we are modifying the meaning of each token based on what we've seen before it in the context, with similar meanings reinforcing each other." At this point in the text, it seems like the kernel smoothing is being applied to each embedding vector in isolation. I don't see why any one y_t vector derived and smoothed from token x_i would be influenced by the nearby tokens in the sequence. When you add the r_t tokens, sure then I see how context matters. But is that the only think that takes into account context?

cousin_it 3y ago

Unlike most articles about machine learning, this one feels like it's written in my language. The section on Lempel-Ziv is cool. Making a large LZ model is probably not very interesting because LZ is too strict, it matches strings exactly. What could be more interesting is something like "jpeg for text", strategically allowing some non-exactness to achieve a reduction in size. Which is probably what LLMs do.

theGnuMe 3y ago
JPEG is FFT which has also been used in DNNs (Fnet encoder/decoder).

linearlayer 3y ago

There is a mistake in the article, right? Multi-headed attention doesn't average together multiple attention heads. Rather it concatenates them and then right-multiplies by a matrix such that the output dimension matches the input dimension. That matrix might learn averaging, but it's not built in a-priori.