I have been enjoying Natural Language Processing with Transformers [1]. It's largely focused on the Huggingface library, but Chapter 3 has a very nice walkthrou

I have been enjoying Natural Language Processing with Transformers [1]. It's largely focused on the Huggingface library, but Chapter 3 has a very nice walkthrough that builds up the encoder portion of an encoder-decoder Transformer from "scratch" (it still uses some primitives found in PyTorch like nn.Embedding). The decoder portion is covered in less depth and they instead refer folks to Karpathy's awesome minGPT [2], which implements a decoder-only (GPT-style) Transformer in ~300 lines of nicely-commented Python+PyTorch code. For a higher-level conceptual view of how Transformers work, you can check out the now-classic "Illustrated Transformer" series [3] and this programmer-oriented explanation (with code in Rust) from someone at Anthropic [4]. [1] https://www.oreilly.com/library/view/natural-language-processing/9781098103231/ https://www.oreilly.com/library/view/natural-language-proces... [2] https://github.com/karpathy/minGPT https://github.com/karpathy/minGPT [3] https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-transformer/ [4] https://blog.nelhage.com/post/transformers-for-software-engineers/ https://blog.nelhage.com/post/transformers-for-software-engi...