3 ms·

this was the deepest dive we ever did into Transformers Math 101 (https://news.ycombinator.com/item?id=35631546 https://news.ycombinator.com/item?id=35631546).

by swyx 3y ago

this was the deepest dive we ever did into Transformers Math 101 (https://news.ycombinator.com/item?id=35631546 https://news.ycombinator.com/item?id=35631546). we mostly cover AI Engineer/inference time things on the pod, but training knowledge is hard won and extremely valuable, and I love it when experts are wiling to distill the rules of thumb for what they have learned. questions welcome!

FanaHOVA 3y ago
Yes! I have a 5 pages doc of prep notes I made, happy to share :)
- rayval 3y ago
  Please do! Thanks
swyx 3y ago
related online discussion when this article came out: from Eleuther: https://twitter.com/AiEleuther/status/1648782486736969728?s=20 https://twitter.com/AiEleuther/status/1648782486736969728?s=... from AI anon: https://twitter.com/nearcyan/status/1662937711156625408?s=20 https://twitter.com/nearcyan/status/1662937711156625408?s=20 Stella Biderman (coauthor and now lead of the Eleuther Foundation, Stella if you're reading this pls come on our pod sometime!!!!) https://twitter.com/BlancheMinerva/status/1648785878288834560?s=20 https://twitter.com/BlancheMinerva/status/164878587828883456...
- stellaathena 3y ago
  I'm quite busy right now, but maybe in late September or October?
photon_lines 3y ago
Sorry for the criticism here, but your linked paper / article does nothing to explain the math behind transformers. You should re-name it something like 'Scaling Transformer Mathematics' or 'The Math Behind Scaling GPUs / Transformers' or whatever more descriptive name there is to describe what you're outlining in your article. 'Transformer Math 101' should at least explain 1) input embeddings 2) (keys, queries, values) and the idea behind the matrix multiplication of these linear / numeric sets of values 3) the Softmax computation in the original paper as well as the relevant matrix transformations which take place 4) dot products ( and how they relate to similarity scoring) 5) feed-forward neural networks and the gradient descent / backpropagation and how they work. There are many many concepts you didn't even touch upon. This is not 'Transformer Math 101' by any means.
- swyx 3y ago
  fwiw i'm not the author of that doc, we just interviewed him, and the hn submission i linked to was also renamed presumably for similar concerns. we do have an "Algorithms 101" episode (in the theme of our Datasets 101 and Benchmarks 101 episode) where we have at least some of your topics lined up

by swyx 3y ago

FanaHOVA 3y ago

Yes! I have a 5 pages doc of prep notes I made, happy to share :)

rayval 3y ago
Please do! Thanks

swyx 3y ago

related online discussion when this article came out: from Eleuther: https://twitter.com/AiEleuther/status/1648782486736969728?s=20 https://twitter.com/AiEleuther/status/1648782486736969728?s=... from AI anon: https://twitter.com/nearcyan/status/1662937711156625408?s=20 https://twitter.com/nearcyan/status/1662937711156625408?s=20 Stella Biderman (coauthor and now lead of the Eleuther Foundation, Stella if you're reading this pls come on our pod sometime!!!!) https://twitter.com/BlancheMinerva/status/1648785878288834560?s=20 https://twitter.com/BlancheMinerva/status/164878587828883456...

stellaathena 3y ago
I'm quite busy right now, but maybe in late September or October?

photon_lines 3y ago

Sorry for the criticism here, but your linked paper / article does nothing to explain the math behind transformers. You should re-name it something like 'Scaling Transformer Mathematics' or 'The Math Behind Scaling GPUs / Transformers' or whatever more descriptive name there is to describe what you're outlining in your article. 'Transformer Math 101' should at least explain 1) input embeddings 2) (keys, queries, values) and the idea behind the matrix multiplication of these linear / numeric sets of values 3) the Softmax computation in the original paper as well as the relevant matrix transformations which take place 4) dot products ( and how they relate to similarity scoring) 5) feed-forward neural networks and the gradient descent / backpropagation and how they work. There are many many concepts you didn't even touch upon. This is not 'Transformer Math 101' by any means.

swyx 3y ago
fwiw i'm not the author of that doc, we just interviewed him, and the hn submission i linked to was also renamed presumably for similar concerns. we do have an "Algorithms 101" episode (in the theme of our Datasets 101 and Benchmarks 101 episode) where we have at least some of your topics lined up