3 ms·
For context: The latest large language models (LLM) skip doing some complex things that previous good models did, and focus on attention. See: "Attention is al
by dwheeler 3y ago
For context: The latest large language models (LLM) skip doing some complex things that previous good models did, and focus on attention. See:
"Attention is all you need"
https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762
- btown 3y agoThis MIT lecture on recurrent neural networks provides a lot of context on why attention was so groundbreaking, and how things were solved before: https://www.youtube.com/watch?v=ySEx_Bqxvvo https://www.youtube.com/watch?v=ySEx_Bqxvvo And for the Attention Is All You Need paper itself, Andrej Karpathy's "GPT From Scratch" talk is a really great walkthrough of how to make this work in practice, all condensed into two hours of live coding: https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY
- hn_throwaway_99 3y agoThat whole MIT course, Intro to Deep Learning, is great and a really nice introduction to the field.