5 ms·
What is a transformer model? (2022)
- sylware 3y agoWeird, still no diagram of a transformer based connectome. Or did I missi it??
- asylteltine 3y agoI think it’s actually much harder to explain this stuff with hand wavy names. I want to see the actual code, how this stuff ACTUALLY works in reality.
- enriquto 3y agoIf you'd rather prefer something readable and explicit, instead of empty handwaving and uml-like diagrams, read "The Transformer model in equations" [0] by John Thickstun [1]. [0] https://johnthickstun.com/docs/transformers.pdf https://johnthickstun.com/docs/transformers.pdf [1] https://johnthickstun.com/docs/ https://johnthickstun.com/docs/
- robotresearcher 3y agoSee also the original paper: https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762
- epistasis 3y agoThe original paper is quite deceptive and hard to understand, IMHO. It relies on jumping between several different figures and mapping between shapes, in addition to guessing at what the unlabeled inputs are. Just a few more labels, making the implicit explicit, would make it far more intelligible. Plus, last time I went through it Im pretty sure that there's either a swap on the order of the three inputs between different figures, or that it's incorrectly diagrammed.
- techbruv 3y agoSome other good resources: [0]: The original paper: https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762 [1]: Full walkthrough for building a GPT from Scratch: https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY [2]: A simple inference only implementation in just NumPy, that's only 60 lines: https://jaykmody.com/blog/gpt-from-scratch/ https://jaykmody.com/blog/gpt-from-scratch/ [3]: Some great visualizations and high-level explanations: http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/ [4]: An implementation that is presented side-by-side with the original paper: https://nlp.seas.harvard.edu/2018/04/03/attention.html https://nlp.seas.harvard.edu/2018/04/03/attention.html
- itissid 3y ago[1] is thoroughly recommended.
- revskill 3y agoRecommended for who ?
- basedbertram 3y agoFrom looking at the video probably someone who has good working knowledge of PyTorch, familiarity with NLP fundamentals and transformers, and somewhat of a working understanding of how GPT works.
- deleted 3y ago[deleted]
- quickthrower2 3y agoMasochists! In a good way! I recommend you do the full course not jump into that video. I did the full course, paused to do some of a university course around lecture 2 to really understand some stuff then came back and finishing it off. Bu the end of you would have done stuff like hand working out back-propagation though sums, broadcasting, batchnorm etc. Fairly intense for a regular programmer!
- cubefox 3y ago(2022) It feels like a joke, but the fact they are mentioning GPT-3 and Megatron-Turing as the hottest new things makes this piece seem so outdated.
- hyperbovine 3y agoThe relevant lifespan of a paper in this field appears to average about three months. Can't really hold that against them.
- jprd 3y agoIf you say "Transformer" and "Megatron" in the same sentence, you have my full-attention for at least enough time to start making a serious observation about ML, or frankly, anything.
- moffkalast 3y agoOnce they realize they can distil and quantize Megatron, I suppose they'll name that one "Starscream"?
- DaiPlusPlus 3y ago> If you say "Transformer" and "Megatron" in the same sentence, you have my full-attention And as they say, attention is all you need
- amelius 3y agoAny self-respecting AI researcher writes self-updating papers.
- edc117 3y agoThere was an excellent thread/discussion about it a while ago: https://news.ycombinator.com/item?id=35977891 https://news.ycombinator.com/item?id=35977891
- semiquaver 3y agoI feel like I have a schematic understanding of how transformer models process text. But I struggle to understand how the same concept could be used with images or other less linear data types. I understand that such data can be vectorized so it’s represented as a point or series of points in a higher dimensional space, but how does such a process capture the essential high level perceptual aspects well enough to be able to replicate large scale features? Or equivalently, how are the dimensions chosen? I have to imagine that simply looking at an image as an RGB pixel sequence would largely miss the point of what it “is” and represents.
- sangnoir 3y ago> how does such a process capture the essential high level perceptual aspects well enough to be able to replicate large scale features Layers and lots of training. Upper layers can capture/recognize large scale features
- mliker 3y agoIf I understand what you're asking, the Transformer isn't initially treating the image as a sequence of pixels like p1, p2, ..., pN. Instead, you can use a convolutional neural network to respect the structure of the image to extract features. Then you use the attention mechanism to pay attention to parts of the image that aren't necessarily close together but that when viewed together, contribute to the classification of an object within the image.
- famouswaffles 3y agoVision Transformers don't use CNNs to extract anything first. (https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929). You could but it's not necessary and it doesn't add anything so it doesn't happen anymore. Vision transformers won't treat the image as a sequence of pixels but that's mostly because doing that gets very expensive very fast. The image is split into patches and the patches have positional embeddings.
- famouswaffles 3y agoJust to add, the first user who replied to you is quite wrong. You can use CNNs to get features first...but it doesn't happen anymore. It's unnecessary and adds nothing. Pixels don't get fed into transformers but that's more expense than anything. Transformers need to understand how each piece relates to every other piece. That gets very costly very fast when the "pieces" are pixels. Images are split into patches instead with positional embeddings. as for how it learns the representations anyway. Well it's not like there's any specific intuition to it. after all, the original authors didn't anticipate the use case in Vision. and the fact that you didn't need CNNs to extract features first didn't really come into light till this paper - https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929 It basically just comes down to lots of layers and training
- simonw 3y agoThis is from March 25, 2022 - over a year old now.
- antonvs 3y agoIt's not Kanye's latest pair of shoes.
- sorenjan 3y agoThere's one thing I haven't quite grasped about transformers. Are the query, key, and value vectors trained per token? Does each token have specific QVK vectors in each head, or does each attention head have one set that are trained using a lot of tokens?
- danielmarkbruce 3y agoThe q,k,v projections are trained, usually per head. But each token goes through the same projection (per head). In some architectures (see the Falcon model) the k, v projections are shared across the heads. During the forward pass a "query" is "created" for each token using the query projection (again, one projection per head, all the tokens run through the same projection). The keys are created using the key projection, the values are created using the dot product of query with the keys for each "token" then projected using the value projection. But again, different models do different things. Some models bring in the positional encoding into the attention calculation rather than adding it in earlier. Practically every combination of things has been tried since that paper was published.
- two_in_one 3y agoThanks, good explanation. More questions. Models have the initial N embeddings passed through, processed by the attention+linear blocks which add something to the input. Sort of memory bus. After the last block we still have array of the same dimensions like N initial embeddings, but they mean something else now. How do we select the next token, based on the last element in that array? Another question, that bus technically doesn't have to have the same width, right? It should be possible, for the same model's size, to trade bus width for the number of heads. Even have sort of U-Net. And the last, on each loop resulting embedding (1) is converted into token, which is being added to the input. Converted to embedding first, it should be possible to just reuse embedding (1), and use token only for user output, right? PS: not sure about every combination tried, we just started. Some of the problems still don't have satisfying solutions. Like hallucinations, online training. Or responsibility.
- 3y ago
- czbond 3y agoIt's more than meets the eye
- karmasimida 3y agoBest teacher of transformer model is ChatGPT itself
- gnicholas 3y agoOne thing I wonder about is where it gets its data about itself from. Did they feed it a bunch of accurate information about itself?
- quickthrower2 3y agoI reckon they did
- gnicholas 3y agoQuite possible! But given how ChatGPT hallucinates, and my general lack of knowledge about LLMs in general and ChatGPT in particular, I would be hesitant to take what it says at face value. I'm especially hesitant to trust anything it says about itself in particular, since much of its specifics are not publicly documented and are essentially unverifiable. I wish there were some way for it to communicate that certain responses about itself were more or less hardcoded.
- smrtinsert 3y agoGreat summary. I'm still working through a class that's just getting to RNN and CNN. Crazy it seems like an obsolete tech already.
- venky180 3y agoWhile it is largely obsolete for practical purposes, learning about them is still valuable as they illustrate the natural evolution in the thought process behind the development of transformers.
- sroussey 3y agoYou learn linked lists in comp sci yet rarely if ever use them. Still, they are a basic concept required to understand trees, DAGs, etc later…
- redox99 3y agoCNNs are still relatively competitive at vision tasks.
- deleted 3y ago[deleted]
- einpoklum 3y agoI think this is a transformer model: https://t.ly/se-M https://t.ly/se-M
- jaimex2 3y agoGreat piece. At the end though there is a concerning bit. > researchers are studying ways to eliminate bias or toxicity if models amplify wrong or harmful language. For example, Stanford created the Center for Research on Foundation Models to explore these issues We've seen numerous times now, censoring/railroading degrades quality. This is why SD 2 was so bad at the human form over 1.4 I get they want a PC AI but this isn't the way.
- nojvek 3y agoSutton’s bitter lesson http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html resonates. General learning + search methods that scale with data + compute win out. Transformers in a sense is one of the most general algorithms able to tackle a whole bunch of domains from audio, video, images, language. I’m sure another better super algorithm will be invented but Transformer is currently what we have.
- VirusNewbie 3y agoIt’s interesting though, isn’t RL even more general?