Tangential, but: the notes that this post refers to are probably not the best way to learn how transformers work. If you want mathematical precision, those note

Tangential, but: the notes that this post refers to are probably not the best way to learn how transformers work. If you want mathematical precision, those notes are based on this paper from DeepMind: https://arxiv.org/abs/2207.09238 https://arxiv.org/abs/2207.09238 The paper provides mathematically precise definitions of all the parts of a transformers, though it's showing its age (ha!) in that it doesn't include some formalizations that are common in, for example, Llama.