5 ms·

What's a good introduction to transformers?

atty 5y ago
If you mean the technical details of attention models, the original paper “Attention is all you need” is not too difficult to read. If you’re more interested in applications, hugging face has a “course” on their website that walks through the high level topics of applying transformers to natural language processing (can’t remember if they cover transformers for other topics).
criticaltinker 5y ago
The original paper that introduced the Transformer architecture is quite accessible and outlines a lot of the history and rational for the design [1]. [1] https://arxiv.org/pdf/1706.03762.pdf https://arxiv.org/pdf/1706.03762.pdf
- mrfusion 5y ago
  I tried that but it seems to gloss over what an encoder, etc actually are. I think I’d do better with pseudo code or a toy example.
  - criticaltinker 5y ago
    Check out The Annotated Transformer, it’s one of my favorite references! It contains straightforward python code side by side with excerpts from the original paper. http://nlp.seas.harvard.edu/2018/04/03/attention.html http://nlp.seas.harvard.edu/2018/04/03/attention.html
  - saynay 5y ago
    The encoder is the neural-net that converts the input to the embedding vector. The decoder is the neural-net that converts that vector into output. What that embedding vector "means" is whatever the entire algorithm has learned it means. For more simplified look at embeddings, I would look at Word2Vec (although, it doesn't involve transformers). It encodes single words, instead of entire phrases, and does so by looking at their relative position to other words while being trained. Embeddings are just vectors, and so you can do math or compare them to other embeddings. The famous example is E(king) - E(man) + E(woman) = E(queen)
    - mrfusion 5y ago
      So you’re saying the encoding could be a neural net OR something like word2vec?
      - saynay 5y ago
        Sorry, didn't see this sooner. Word2Vec was just an example of an easier-to-understand embedding (it is actually frequently used as an input to these language models, actually). The process of encoding the input data stream to an embedding is done by a NN. The part that tripped me up when I was first trying to understand it is that the encoding is not like encoding elsewhere in compsci. It isn't trying to convert the input to a well-known encoding format or anything. The "encoder" is just the name of the neural-net that is converting the big, multi-dimensional input into a single-dimension, fixed-length vector. The "decoder" is just the neural-net that converts it to output. That fixed-length vector is entirely meaningless to anything other than the two neural-nets trained to generate/consume it.
        mrfusion 5y ago
        Thanks. That’s helping the clear it up. So what’s the point of the encoding and decoding steps? Then the transformer is just another neural net in between that does some funny business with attention?
        saynay 5y ago
        The transformer is the neural net in the encode / decode steps. For something like a language model, it takes an input phrase, encodes it to something that has "meaning" to the net, then decodes it into an output phrase. The "attention" mechanism learns how much attention to give to each part of the input, anywhere in it. In languages, the previous common mechanisms (recurrent networks), would only look backwards, with decreasing weight for things further away.
  - matt123456789 5y ago
    The attention mechanism, specifically the confusing bits about the Q/K/V layers, are all single linear layers with bias followed by a nonlinearity. Each of Q, K, and V are just square matrices with edge length the same as the “dimension” of the network (which you usually see denoted as “d”). So if you have an input sentence of 10 words, it gets embedded into a 10 x d space by looking up each word’s d-dimensional representation (from a |V| x d matrix). Add in some static position-dependent embedding, which is usually a function of the time index, so that the model can use position information and call the result E_0. Multiply E_0 by your d x d matrix of keys K and set aside as K_0. Do the same for your d x d query matrix Q and call the result Q_0. (The result in both cases is shape 10 x d). Take (Q • K’) which results in a 10 x 10 matrix where entry i,j denotes the amount that vector i attends to vector j. Squash this with a softmax along the time dimension* and set it aside as “attentions”. Create V_0 by running your inputs E_0 through the d x d layer of values V. Finally, calculate the output of the attention layer as (attentions) * V_0. This is just a linear cimbination of the values matrix V_0, where the coefficients are the softmaxed attention values from earlier. This is exactly what’s going on in Eq 1 from the “Attention is all you need” paper. A transformer block is usually the above operation, and then it immediately runs the result through a high-dimensional MLP ending with a layer with dimension d. And it will usually use multi-head attention which splits the input d-dimensional representation into m splits along the d axis (so that you end up with m 10 x m/d vectors) and then does the above calculation on each of the m inputs using layer- and head-specific QKV matrices, and then recombines all the results along the d axis to recover the 10 x d representation. It can be confusing when you read code that does this because it’s all vectorized and the splitting and head-specific matrices are implicit. * after dividing by the square root of d. this reduces absolute value of the inputs to the softmax and mitigates an issue that softmax has with gradients on large inputs becoming 0 on limited precision floating points.
    - mrfusion 5y ago
      Wow thanks. That’s getting clearer. A lot of those steps come off as arbitrary but I guess there’s a good reason behind each step.
lalaithion 5y ago
Not sure if this is a good introduction, but a good second paper to read is https://arxiv.org/abs/2106.06981 https://arxiv.org/abs/2106.06981 You can think of finite state machines as being two functions: f(input, state) = output, and g(input, state) = next_state. (Traditional FSMs have 3 ‘output’ states, basically ‘terminated - success’, ‘terminated - failure’, and ‘still working’, but in theory it makes sense to fully generalize it). If you think about plain neural networks as approximating arbitrary functions f(input) = output, then recurrent neural networks are “continuous state machines”, where you have the same two functions f(input, state) = output, and g(input, state) = next_state, except instead of being finite symbols, they’re continuous points in N dimensional space. This, at least to me, clarifies why recurrent neural networks work on simple and short time-based problems, but can’t efficiently generalize to complex problems—they’re just FSMs! The paper I linked above provides a similar high-level computational analogy to how transformers work.
beefman 5y ago
Transformers from Scratch link: https://e2eml.school/transformers.html https://e2eml.school/transformers.html discussion here: https://news.ycombinator.com/item?id=29315107 https://news.ycombinator.com/item?id=29315107
moffkalast 5y ago
Transformers (2007)
- timy2shoes 5y ago
  I prefer to go to the original source, specifically The Transformers (1984-87).
visarga 5y ago
For accessibility I recommend Yannic Kilcher video review of "Attention Is All You Need" https://www.youtube.com/watch?v=iDulhoQ2pro https://www.youtube.com/watch?v=iDulhoQ2pro Yannic has been making about 62 other transformer paper reviews since. You can find the usual suspects. https://www.youtube.com/watch?v=u1_qMdb0kYU&list=PL1v8zpldgH3pQwRz1FORZdChMaNZaR3pu https://www.youtube.com/watch?v=u1_qMdb0kYU&list=PL1v8zpldgH...
tchalla 5y ago
I have seen a lot of introduction that explains the mechanics. However, I haven't seen one that explains the intuition of the hypothesis on why it works.
dpflan 5y ago
Another resource: "The Illustrated Transformer" - https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-transformer/ - HN post for the article: https://news.ycombinator.com/item?id=18351674 https://news.ycombinator.com/item?id=18351674
d4rkp4ttern 5y ago
IMHO this is a very good intuitive intro to transformers by Peter Bloem. It does a great job of starting with basic intuition and gradually adding in complexity. Once you read it you’ll have a good mental picture of how self attention works. Blogs allow this type of informal, relaxed style of explanation that academic papers can’t afford (page number submission limits etc ) http://peterbloem.nl/blog/transformers http://peterbloem.nl/blog/transformers

atty 5y ago

If you mean the technical details of attention models, the original paper “Attention is all you need” is not too difficult to read. If you’re more interested in applications, hugging face has a “course” on their website that walks through the high level topics of applying transformers to natural language processing (can’t remember if they cover transformers for other topics).

criticaltinker 5y ago

The original paper that introduced the Transformer architecture is quite accessible and outlines a lot of the history and rational for the design [1]. [1] https://arxiv.org/pdf/1706.03762.pdf https://arxiv.org/pdf/1706.03762.pdf

mrfusion 5y ago
I tried that but it seems to gloss over what an encoder, etc actually are. I think I’d do better with pseudo code or a toy example.
- criticaltinker 5y ago
  Check out The Annotated Transformer, it’s one of my favorite references! It contains straightforward python code side by side with excerpts from the original paper. http://nlp.seas.harvard.edu/2018/04/03/attention.html http://nlp.seas.harvard.edu/2018/04/03/attention.html
- saynay 5y ago
  The encoder is the neural-net that converts the input to the embedding vector. The decoder is the neural-net that converts that vector into output. What that embedding vector "means" is whatever the entire algorithm has learned it means. For more simplified look at embeddings, I would look at Word2Vec (although, it doesn't involve transformers). It encodes single words, instead of entire phrases, and does so by looking at their relative position to other words while being trained. Embeddings are just vectors, and so you can do math or compare them to other embeddings. The famous example is E(king) - E(man) + E(woman) = E(queen)
  - mrfusion 5y ago
    So you’re saying the encoding could be a neural net OR something like word2vec?
    - saynay 5y ago
      Sorry, didn't see this sooner. Word2Vec was just an example of an easier-to-understand embedding (it is actually frequently used as an input to these language models, actually). The process of encoding the input data stream to an embedding is done by a NN. The part that tripped me up when I was first trying to understand it is that the encoding is not like encoding elsewhere in compsci. It isn't trying to convert the input to a well-known encoding format or anything. The "encoder" is just the name of the neural-net that is converting the big, multi-dimensional input into a single-dimension, fixed-length vector. The "decoder" is just the neural-net that converts it to output. That fixed-length vector is entirely meaningless to anything other than the two neural-nets trained to generate/consume it.
      - mrfusion 5y ago
        Thanks. That’s helping the clear it up. So what’s the point of the encoding and decoding steps? Then the transformer is just another neural net in between that does some funny business with attention?
        saynay 5y ago
        The transformer is the neural net in the encode / decode steps. For something like a language model, it takes an input phrase, encodes it to something that has "meaning" to the net, then decodes it into an output phrase. The "attention" mechanism learns how much attention to give to each part of the input, anywhere in it. In languages, the previous common mechanisms (recurrent networks), would only look backwards, with decreasing weight for things further away.
- matt123456789 5y ago
  The attention mechanism, specifically the confusing bits about the Q/K/V layers, are all single linear layers with bias followed by a nonlinearity. Each of Q, K, and V are just square matrices with edge length the same as the “dimension” of the network (which you usually see denoted as “d”). So if you have an input sentence of 10 words, it gets embedded into a 10 x d space by looking up each word’s d-dimensional representation (from a |V| x d matrix). Add in some static position-dependent embedding, which is usually a function of the time index, so that the model can use position information and call the result E_0. Multiply E_0 by your d x d matrix of keys K and set aside as K_0. Do the same for your d x d query matrix Q and call the result Q_0. (The result in both cases is shape 10 x d). Take (Q • K’) which results in a 10 x 10 matrix where entry i,j denotes the amount that vector i attends to vector j. Squash this with a softmax along the time dimension* and set it aside as “attentions”. Create V_0 by running your inputs E_0 through the d x d layer of values V. Finally, calculate the output of the attention layer as (attentions) * V_0. This is just a linear cimbination of the values matrix V_0, where the coefficients are the softmaxed attention values from earlier. This is exactly what’s going on in Eq 1 from the “Attention is all you need” paper. A transformer block is usually the above operation, and then it immediately runs the result through a high-dimensional MLP ending with a layer with dimension d. And it will usually use multi-head attention which splits the input d-dimensional representation into m splits along the d axis (so that you end up with m 10 x m/d vectors) and then does the above calculation on each of the m inputs using layer- and head-specific QKV matrices, and then recombines all the results along the d axis to recover the 10 x d representation. It can be confusing when you read code that does this because it’s all vectorized and the splitting and head-specific matrices are implicit. * after dividing by the square root of d. this reduces absolute value of the inputs to the softmax and mitigates an issue that softmax has with gradients on large inputs becoming 0 on limited precision floating points.
  - mrfusion 5y ago
    Wow thanks. That’s getting clearer. A lot of those steps come off as arbitrary but I guess there’s a good reason behind each step.

lalaithion 5y ago

Not sure if this is a good introduction, but a good second paper to read is https://arxiv.org/abs/2106.06981 https://arxiv.org/abs/2106.06981 You can think of finite state machines as being two functions: f(input, state) = output, and g(input, state) = next_state. (Traditional FSMs have 3 ‘output’ states, basically ‘terminated - success’, ‘terminated - failure’, and ‘still working’, but in theory it makes sense to fully generalize it). If you think about plain neural networks as approximating arbitrary functions f(input) = output, then recurrent neural networks are “continuous state machines”, where you have the same two functions f(input, state) = output, and g(input, state) = next_state, except instead of being finite symbols, they’re continuous points in N dimensional space. This, at least to me, clarifies why recurrent neural networks work on simple and short time-based problems, but can’t efficiently generalize to complex problems—they’re just FSMs! The paper I linked above provides a similar high-level computational analogy to how transformers work.

beefman 5y ago

Transformers from Scratch link: https://e2eml.school/transformers.html https://e2eml.school/transformers.html discussion here: https://news.ycombinator.com/item?id=29315107 https://news.ycombinator.com/item?id=29315107

moffkalast 5y ago

Transformers (2007)

timy2shoes 5y ago
I prefer to go to the original source, specifically The Transformers (1984-87).

visarga 5y ago

For accessibility I recommend Yannic Kilcher video review of "Attention Is All You Need" https://www.youtube.com/watch?v=iDulhoQ2pro https://www.youtube.com/watch?v=iDulhoQ2pro Yannic has been making about 62 other transformer paper reviews since. You can find the usual suspects. https://www.youtube.com/watch?v=u1_qMdb0kYU&list=PL1v8zpldgH3pQwRz1FORZdChMaNZaR3pu https://www.youtube.com/watch?v=u1_qMdb0kYU&list=PL1v8zpldgH...

tchalla 5y ago

I have seen a lot of introduction that explains the mechanics. However, I haven't seen one that explains the intuition of the hypothesis on why it works.

dpflan 5y ago

Another resource: "The Illustrated Transformer" - https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-transformer/ - HN post for the article: https://news.ycombinator.com/item?id=18351674 https://news.ycombinator.com/item?id=18351674

d4rkp4ttern 5y ago

IMHO this is a very good intuitive intro to transformers by Peter Bloem. It does a great job of starting with basic intuition and gradually adding in complexity. Once you read it you’ll have a good mental picture of how self attention works. Blogs allow this type of informal, relaxed style of explanation that academic papers can’t afford (page number submission limits etc ) http://peterbloem.nl/blog/transformers http://peterbloem.nl/blog/transformers