5 ms·
An early explainer of transformers, which is a quicker read, that I found very useful when they were still new to me, is The Illustrated Transformer[1], by Jay
by dsubburam 3y ago
An early explainer of transformers, which is a quicker read, that I found very useful when they were still new to me, is The Illustrated Transformer[1], by Jay Alammar.
A more recent academic but high-level explanation of transformers, very good for detail on the different flow flavors (e.g. encoder-decoder vs decoder only), is Formal Algorithms for Transformers[2], from DeepMind.
[1] https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-transformer/
[2] https://arxiv.org/abs/2207.09238 https://arxiv.org/abs/2207.09238
- driscoll42 3y agoThe Illustrated Transformer is fantastic, but I would suggest that those going into it really should read the previous articles in the series to get a foundation to understand it more, plus later articles that go into GPT and BERT, here's the list: A Visual and Interactive Guide to the Basics of Neural Networks - https://jalammar.github.io/visual-interactive-guide-basics-neural-networks/ https://jalammar.github.io/visual-interactive-guide-basics-n... A Visual And Interactive Look at Basic Neural Network Math - https://jalammar.github.io/feedforward-neural-networks-visual-interactive/ https://jalammar.github.io/feedforward-neural-networks-visua... Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) - https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/ https://jalammar.github.io/visualizing-neural-machine-transl... The Illustrated Transformer - https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-transformer/ The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) - https://jalammar.github.io/illustrated-bert/ https://jalammar.github.io/illustrated-bert/ The Illustrated GPT-2 (Visualizing Transformer Language Models) - https://jalammar.github.io/illustrated-gpt2/ https://jalammar.github.io/illustrated-gpt2/ How GPT3 Works - Visualizations and Animations - https://jalammar.github.io/how-gpt3-works-visualizations-animations/ https://jalammar.github.io/how-gpt3-works-visualizations-ani... The Illustrated Retrieval Transformer - https://jalammar.github.io/illustrated-retrieval-transformer/ https://jalammar.github.io/illustrated-retrieval-transformer... The Illustrated Stable Diffusion - https://jalammar.github.io/illustrated-stable-diffusion/ https://jalammar.github.io/illustrated-stable-diffusion/ If you want to learn how to code them, this book is great: https://d2l.ai/chapter_attention-mechanisms-and-transformers/index.html https://d2l.ai/chapter_attention-mechanisms-and-transformers...
- mkw5053 3y agoThanks for sharing!
- behnamoh 3y agoCan anyone please say how much value there is in learning the fundamentals of LLMs for someone who uses them in practice?
- m3kw9 3y agoA little understand would give you why certain prompts work or don’t work. A high level should do. It could help you make better prompts or trouble shoot, although you don’t need it for 80% of the cases
- noobcoder 3y agoI remember looking into this article. It was really helpful for me to understand transformers. Although the OP's article is detailed, this one is concise. Here's the link: https://blue-season.github.io/transformer-in-5-minutes https://blue-season.github.io/transformer-in-5-minutes
- abhgh 3y agoI would also recommend Pascal Poupart's talk [1] on Attention and Transformers. Doesn't seem to be cited often, but its quite well done and seeing him work out some of the details using chalk and board is very reassuring. [1] https://www.youtube.com/watch?v=OyFJWRnt_AY https://www.youtube.com/watch?v=OyFJWRnt_AY