3 ms·

As with most tutorials on Transformers, this one leaves out some essential details: - how are the input encodings generated? - what is in those position vecto

by ttul 4y ago

As with most tutorials on Transformers, this one leaves out some essential details: - how are the input encodings generated? - what is in those position vectors? - how are the attention vectors learned? The answer is that these things are all learned as the network is trained; the whole thing is one “thing”. The concept that is most important in understanding neural networks generally is that they start out as just a bunch of random numbers and then the numbers are gradually adjusted until the outputs converge closely enough on the desired loss. I recommend watching Karpathy’s YouTube video where he codes up a Transformer from scratch. It’s the best way to understand these beasts.

tomcam 4y ago
Thank you. This one? https://youtu.be/kCc8FmEb1nY https://youtu.be/kCc8FmEb1nY
- totoglazer 4y ago
  I’m sure that’s the one being referred to. It’s great.
- leobg 4y ago
  Is that an actual transformer, though? Like with encoder and decoder layers? That’s the part I never truly understood. Or is it “just” an example of a neural network? Thanks!
  - tarvaina 4y ago
    Not all transformers have separate encoders and decoders. GPTs, for instance, only have the equivalents of decoder layers of the original transformer paper, but they are still considered transformers. Karpathy’s video shows an actual GPT-style transformer. I think a neural network can be considered a transformer if it contains a stack of attention blocks as its core mechanism.
  - HarHarVeryFunny 4y ago
    The original transformer, from the "attention is all you need paper" had an encoder-decoder architecture because it was designed for language translation use where you are mapping one sequence to another, and are able to use both the preceding and following context of words when performing this mapping. The encoder utilizes the forward context. In contrast to seq-2-seq use, for generative language models such as ChatGPT you only have access to preceding (not forward) context in order to decide what to generate next, so the encoder part of the architecture is not applicable and a decoder-only transformer is used.
seydor 4y ago
note that embedding and positional vectors are fixed and not trained with the transformer. The attention vectors are also computed in each step, but what is learned is the transformation of inputs to qvk vectors and the fully connected layers. There are some great people on youtube who explain the series of steps. I think this is the most comprehensive: https://www.youtube.com/watch?v=Nw_PJdmydZY https://www.youtube.com/watch?v=Nw_PJdmydZY
- ttul 4y ago
  Thank you so much for setting me straight. CodeEmporium’s 10-part explainer is so great.
HarHarVeryFunny 4y ago
Yes, that's increasingly the case, but there's no fundamental reason for nets to be trained end-to-end as a single entity. Going back a few years it used to be quite common for people to use fixed word embeddings such as word2vec rather than learning them, and for image classification to take an ImageNet-pretrained general purpose model, then freeze the lower convolutional feature-detector layers and only train a new model "head" for more specialized use. End-to-end learnt embeddings are going to be more optimal though, and in the context of these massive models the computational cost of training them is a drop in the bucket!

by ttul 4y ago

tomcam 4y ago

Thank you. This one? https://youtu.be/kCc8FmEb1nY https://youtu.be/kCc8FmEb1nY

totoglazer 4y ago
I’m sure that’s the one being referred to. It’s great.
leobg 4y ago
Is that an actual transformer, though? Like with encoder and decoder layers? That’s the part I never truly understood. Or is it “just” an example of a neural network? Thanks!
- tarvaina 4y ago
  Not all transformers have separate encoders and decoders. GPTs, for instance, only have the equivalents of decoder layers of the original transformer paper, but they are still considered transformers. Karpathy’s video shows an actual GPT-style transformer. I think a neural network can be considered a transformer if it contains a stack of attention blocks as its core mechanism.
- HarHarVeryFunny 4y ago
  The original transformer, from the "attention is all you need paper" had an encoder-decoder architecture because it was designed for language translation use where you are mapping one sequence to another, and are able to use both the preceding and following context of words when performing this mapping. The encoder utilizes the forward context. In contrast to seq-2-seq use, for generative language models such as ChatGPT you only have access to preceding (not forward) context in order to decide what to generate next, so the encoder part of the architecture is not applicable and a decoder-only transformer is used.

seydor 4y ago

note that embedding and positional vectors are fixed and not trained with the transformer. The attention vectors are also computed in each step, but what is learned is the transformation of inputs to qvk vectors and the fully connected layers. There are some great people on youtube who explain the series of steps. I think this is the most comprehensive: https://www.youtube.com/watch?v=Nw_PJdmydZY https://www.youtube.com/watch?v=Nw_PJdmydZY

ttul 4y ago
Thank you so much for setting me straight. CodeEmporium’s 10-part explainer is so great.

HarHarVeryFunny 4y ago

Yes, that's increasingly the case, but there's no fundamental reason for nets to be trained end-to-end as a single entity. Going back a few years it used to be quite common for people to use fixed word embeddings such as word2vec rather than learning them, and for image classification to take an ImageNet-pretrained general purpose model, then freeze the lower convolutional feature-detector layers and only train a new model "head" for more specialized use. End-to-end learnt embeddings are going to be more optimal though, and in the context of these massive models the computational cost of training them is a drop in the bucket!