2 ms·
Perhaps some of the really old texts by Kahonen, Carver Mead, etc.? For more modern material, there are a few new good books on Transformers. Transformers are
by mark_l_watson 5y ago
Perhaps some of the really old texts by Kahonen, Carver Mead, etc.?
For more modern material, there are a few new good books on Transformers. Transformers are interesting because they were designed for efficiency: layers the same size, encoding both data and time sequencing information in each sample (so recurrent NNs aren’t required), etc.
- pfd1986 5y agoMind sharing some titles?
- moyix 5y agoI have been enjoying Natural Language Processing with Transformers [1]. It's largely focused on the Huggingface library, but Chapter 3 has a very nice walkthrough that builds up the encoder portion of an encoder-decoder Transformer from "scratch" (it still uses some primitives found in PyTorch like nn.Embedding). The decoder portion is covered in less depth and they instead refer folks to Karpathy's awesome minGPT [2], which implements a decoder-only (GPT-style) Transformer in ~300 lines of nicely-commented Python+PyTorch code. For a higher-level conceptual view of how Transformers work, you can check out the now-classic "Illustrated Transformer" series [3] and this programmer-oriented explanation (with code in Rust) from someone at Anthropic [4]. [1] https://www.oreilly.com/library/view/natural-language-processing/9781098103231/ https://www.oreilly.com/library/view/natural-language-proces... [2] https://github.com/karpathy/minGPT https://github.com/karpathy/minGPT [3] https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-transformer/ [4] https://blog.nelhage.com/post/transformers-for-software-engineers/ https://blog.nelhage.com/post/transformers-for-software-engi...