9 ms·

The Mathematics of Training LLMs

199 points by FanaHOVA 3y ago

swyx 3y ago
this was the deepest dive we ever did into Transformers Math 101 (https://news.ycombinator.com/item?id=35631546 https://news.ycombinator.com/item?id=35631546). we mostly cover AI Engineer/inference time things on the pod, but training knowledge is hard won and extremely valuable, and I love it when experts are wiling to distill the rules of thumb for what they have learned. questions welcome!
- FanaHOVA 3y ago
  Yes! I have a 5 pages doc of prep notes I made, happy to share :)
  - rayval 3y ago
    Please do! Thanks
- swyx 3y ago
  related online discussion when this article came out: from Eleuther: https://twitter.com/AiEleuther/status/1648782486736969728?s=20 https://twitter.com/AiEleuther/status/1648782486736969728?s=... from AI anon: https://twitter.com/nearcyan/status/1662937711156625408?s=20 https://twitter.com/nearcyan/status/1662937711156625408?s=20 Stella Biderman (coauthor and now lead of the Eleuther Foundation, Stella if you're reading this pls come on our pod sometime!!!!) https://twitter.com/BlancheMinerva/status/1648785878288834560?s=20 https://twitter.com/BlancheMinerva/status/164878587828883456...
  - stellaathena 3y ago
    I'm quite busy right now, but maybe in late September or October?
- photon_lines 3y ago
  Sorry for the criticism here, but your linked paper / article does nothing to explain the math behind transformers. You should re-name it something like 'Scaling Transformer Mathematics' or 'The Math Behind Scaling GPUs / Transformers' or whatever more descriptive name there is to describe what you're outlining in your article. 'Transformer Math 101' should at least explain 1) input embeddings 2) (keys, queries, values) and the idea behind the matrix multiplication of these linear / numeric sets of values 3) the Softmax computation in the original paper as well as the relevant matrix transformations which take place 4) dot products ( and how they relate to similarity scoring) 5) feed-forward neural networks and the gradient descent / backpropagation and how they work. There are many many concepts you didn't even touch upon. This is not 'Transformer Math 101' by any means.
  - swyx 3y ago
    fwiw i'm not the author of that doc, we just interviewed him, and the hn submission i linked to was also renamed presumably for similar concerns. we do have an "Algorithms 101" episode (in the theme of our Datasets 101 and Benchmarks 101 episode) where we have at least some of your topics lined up
kitanata 3y ago
[dead]
mchiang 3y ago
Thanks for sharing this. How do you think the local LLM movement will evolve? Especially as in the post, you mentioned startups and VCs both hoarding GPUs to attract talent. There seems to be a good demand behind tools like llama.cpp or ollama (https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama) to run models locally. Maybe as the local runners become more efficient, we'll start seeing more trainings for smaller models or fine-tuning done locally? I too am still trying to wrap my head around this.
- CuriouslyC 3y ago
  The current generation of models (Llama/Llama2) seem to pass a threshold of "good enough" for the majority of use cases at 60b+ parameters. Quantized 30b models that can run in 24gb of GPU VRAM are good enough for many applications but definitely show their limitations frequently. It is likely that we will eventually see good fine tunes for Llama 30b that produce usable output for code/other challenging domains, but until we get GPUs with 48g+ VRAM we're going to have to make do with general models that aren't great at anything, and fine tunes that only do one very narrow thing well.
- swyx 3y ago
  apart from the obvious GGML, we've done podcasts with both MLC/TQ Chen (https://www.latent.space/p/llms-everywhere https://www.latent.space/p/llms-everywhere) and Tiny/George Hotz (https://www.latent.space/p/geohot https://www.latent.space/p/geohot) who are building out more tooling for the Local LLM space! there's actually already a ton of interest, and arguably if you go by the huggingface model hub it's actually a very well developed ecosystem.. just that a lot of the usecases tend to be NSFW oriented. still, i'm looking to do more podcasts in this space, please let me know if any good guests come to mind.
- FanaHOVA 3y ago
  Training smaller models can be really compute intensive (a 7B models should get trained on 1.4T tokens to follow the "LLaMA laws"). So that would be C = 6 * 1.4T * 7B = 58.8T FLOP-seconds. That's 1/5th the compute of GPT3 for example, but it's still a lot. We asked Quentin to do a similar post but for fine tuning math; that's still a very underexplored space. (Not to self plug too much, but this is exactly what last episode's with Tianqi Chen was about if you're interested :) https://www.latent.space/p/llms-everywhere#details https://www.latent.space/p/llms-everywhere#details
babelfish 3y ago
Why is the constant in the formula 6? In the transcript I see a mention of "6 tokens per parameter", but I'm not clear on why that is.
- swyx 3y ago
  if you click through to the source doc and we talk about it on the pod, its basically 2 for forward pass, 4 for backward pass
- FanaHOVA 3y ago
  It's 2 for forward pass, 4 for backward pass (2PD + 4PD = 6PD), but we didn't cover why it's 2 and 4 since it was a very deep rabbit hole. Maybe he'll do a separate post for that in the future!
  - arugulum 3y ago
    If you want a speedrun explanation for how we get to "2": In the limit of model scaling, context size doesn't matter (yes, forget about the quadratic attention), most of the compute is in the linear layers, which boil down to matrix multiplies. Consider a single matrix of size [T,d] multiplied by weight of size [d,d], the compute needed for a matrix multiplication is approximately 2Td^2 (2 coming from multiply + add). Swap T out with D for your whole dataset in tokens, d^2 is the number of parameters in a single linear layer so scale up your model to P, and you've got 2PD. Even shorter: The 2 comes from the multiply-add
  - float-trip 3y ago
    There's a breakdown here for anyone interested (ctrl+f "weight flops for") https://medium.com/@dzmitrybahdanau/the-flops-calculus-of-language-model-training-3b19c1f025e4 https://medium.com/@dzmitrybahdanau/the-flops-calculus-of-la...
tysam_and 3y ago
I think this is a lot of the mathematics of scaling LLM training. Which is quite important! One fundamental requirement though for any machine learning engineer working on these kinds of systems is https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf https://people.math.harvard.edu/~ctm/home/text/others/shanno.... I do not want to be entirely hypocritical as I am still ingesting this theory myself (started several years ago!), but I've found it _absolutely crucial_ in working in ML, as it implicitly informs every single decision you make when designing, deploying, and scaling neural networks. Without it, I feel the field turns into an empirical "stabby stab-around in the dark" kind of game, which very much has its dopamine highs and lows, but ala Sutton, does not scale very well in the long-run. ;P
- jackblemming 3y ago
  Please list a single decision you've made that was directly influenced by Shannon's paper, and no I do not mean something you post hoc rationalized.
  - jdkoeck 3y ago
    Good challenge, I hope we will get a response.
    - tysam_and 3y ago
      Your wish hath been granted! <3 :))))
  - tysam_and 3y ago
    Sure. Basically everything in https://github.com/tysam-code/hlb-CIFAR10 https://github.com/tysam-code/hlb-CIFAR10 was directly founded on concepts shared in the above paper, down to the coding, commenting, and layout styles (hence why I advocate so strongly for it as a requirement for ML. The empirical benefits are clear to me). Before I sat down and wrote my first line, I spent a very long time thinking about how to optimize the repo. Not just in terms of information flow during training, but how the code was laid out (minimize the expected value of deltas for changes from a superset of possible code changes), to even the explanatory comments (ratio of space vs mental effort to decode the repo for experienced vs inexperienced developers). I really want it to be a good exemplary model of a different, more scalable, and more efficient way of conducting small-scale (and potentially resource-constrained) research. To do that, you have to maximize information efficiency at every stage of the pipeline, including temporally (!!!!). It's not perfect, but I've used info theory as a strong guiding light for that repo. There's more to say here, but it's a long conversation about the expected utility of doing research a few different kinds of ways.
xchip 3y ago
Ambiguous title, it should be "memory requirements for training LLMs"
- d4rkp4ttern 3y ago
  Thanks. This is why I read HN comments before the article :)
Havoc 3y ago
How feasible is it to train a toy model / very niche model on high end consumer hardware. Ie 24gb gpu? From scratch not tuning
- sciolist 3y ago
  It depends on the number of parameters you're going for - a billion parameters is going to be out of reach from scratch, but you could train some in the millions.
andirk 3y ago
These headlines read like owning a Tamagotchi virtual pet. Will it become popular to have ones own LLM, name it, and battle others!?

199 points by FanaHOVA 3y ago

swyx 3y ago

this was the deepest dive we ever did into Transformers Math 101 (https://news.ycombinator.com/item?id=35631546 https://news.ycombinator.com/item?id=35631546). we mostly cover AI Engineer/inference time things on the pod, but training knowledge is hard won and extremely valuable, and I love it when experts are wiling to distill the rules of thumb for what they have learned. questions welcome!

FanaHOVA 3y ago
Yes! I have a 5 pages doc of prep notes I made, happy to share :)
- rayval 3y ago
  Please do! Thanks
swyx 3y ago
related online discussion when this article came out: from Eleuther: https://twitter.com/AiEleuther/status/1648782486736969728?s=20 https://twitter.com/AiEleuther/status/1648782486736969728?s=... from AI anon: https://twitter.com/nearcyan/status/1662937711156625408?s=20 https://twitter.com/nearcyan/status/1662937711156625408?s=20 Stella Biderman (coauthor and now lead of the Eleuther Foundation, Stella if you're reading this pls come on our pod sometime!!!!) https://twitter.com/BlancheMinerva/status/1648785878288834560?s=20 https://twitter.com/BlancheMinerva/status/164878587828883456...
- stellaathena 3y ago
  I'm quite busy right now, but maybe in late September or October?
photon_lines 3y ago
Sorry for the criticism here, but your linked paper / article does nothing to explain the math behind transformers. You should re-name it something like 'Scaling Transformer Mathematics' or 'The Math Behind Scaling GPUs / Transformers' or whatever more descriptive name there is to describe what you're outlining in your article. 'Transformer Math 101' should at least explain 1) input embeddings 2) (keys, queries, values) and the idea behind the matrix multiplication of these linear / numeric sets of values 3) the Softmax computation in the original paper as well as the relevant matrix transformations which take place 4) dot products ( and how they relate to similarity scoring) 5) feed-forward neural networks and the gradient descent / backpropagation and how they work. There are many many concepts you didn't even touch upon. This is not 'Transformer Math 101' by any means.
- swyx 3y ago
  fwiw i'm not the author of that doc, we just interviewed him, and the hn submission i linked to was also renamed presumably for similar concerns. we do have an "Algorithms 101" episode (in the theme of our Datasets 101 and Benchmarks 101 episode) where we have at least some of your topics lined up

kitanata 3y ago

[dead]

mchiang 3y ago

Thanks for sharing this. How do you think the local LLM movement will evolve? Especially as in the post, you mentioned startups and VCs both hoarding GPUs to attract talent. There seems to be a good demand behind tools like llama.cpp or ollama (https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama) to run models locally. Maybe as the local runners become more efficient, we'll start seeing more trainings for smaller models or fine-tuning done locally? I too am still trying to wrap my head around this.

CuriouslyC 3y ago
The current generation of models (Llama/Llama2) seem to pass a threshold of "good enough" for the majority of use cases at 60b+ parameters. Quantized 30b models that can run in 24gb of GPU VRAM are good enough for many applications but definitely show their limitations frequently. It is likely that we will eventually see good fine tunes for Llama 30b that produce usable output for code/other challenging domains, but until we get GPUs with 48g+ VRAM we're going to have to make do with general models that aren't great at anything, and fine tunes that only do one very narrow thing well.
swyx 3y ago
apart from the obvious GGML, we've done podcasts with both MLC/TQ Chen (https://www.latent.space/p/llms-everywhere https://www.latent.space/p/llms-everywhere) and Tiny/George Hotz (https://www.latent.space/p/geohot https://www.latent.space/p/geohot) who are building out more tooling for the Local LLM space! there's actually already a ton of interest, and arguably if you go by the huggingface model hub it's actually a very well developed ecosystem.. just that a lot of the usecases tend to be NSFW oriented. still, i'm looking to do more podcasts in this space, please let me know if any good guests come to mind.
FanaHOVA 3y ago
Training smaller models can be really compute intensive (a 7B models should get trained on 1.4T tokens to follow the "LLaMA laws"). So that would be C = 6 * 1.4T * 7B = 58.8T FLOP-seconds. That's 1/5th the compute of GPT3 for example, but it's still a lot. We asked Quentin to do a similar post but for fine tuning math; that's still a very underexplored space. (Not to self plug too much, but this is exactly what last episode's with Tianqi Chen was about if you're interested :) https://www.latent.space/p/llms-everywhere#details https://www.latent.space/p/llms-everywhere#details

babelfish 3y ago

Why is the constant in the formula 6? In the transcript I see a mention of "6 tokens per parameter", but I'm not clear on why that is.

swyx 3y ago
if you click through to the source doc and we talk about it on the pod, its basically 2 for forward pass, 4 for backward pass
FanaHOVA 3y ago
It's 2 for forward pass, 4 for backward pass (2PD + 4PD = 6PD), but we didn't cover why it's 2 and 4 since it was a very deep rabbit hole. Maybe he'll do a separate post for that in the future!
- arugulum 3y ago
  If you want a speedrun explanation for how we get to "2": In the limit of model scaling, context size doesn't matter (yes, forget about the quadratic attention), most of the compute is in the linear layers, which boil down to matrix multiplies. Consider a single matrix of size [T,d] multiplied by weight of size [d,d], the compute needed for a matrix multiplication is approximately 2Td^2 (2 coming from multiply + add). Swap T out with D for your whole dataset in tokens, d^2 is the number of parameters in a single linear layer so scale up your model to P, and you've got 2PD. Even shorter: The 2 comes from the multiply-add
- float-trip 3y ago
  There's a breakdown here for anyone interested (ctrl+f "weight flops for") https://medium.com/@dzmitrybahdanau/the-flops-calculus-of-language-model-training-3b19c1f025e4 https://medium.com/@dzmitrybahdanau/the-flops-calculus-of-la...

tysam_and 3y ago

I think this is a lot of the mathematics of scaling LLM training. Which is quite important! One fundamental requirement though for any machine learning engineer working on these kinds of systems is https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf https://people.math.harvard.edu/~ctm/home/text/others/shanno.... I do not want to be entirely hypocritical as I am still ingesting this theory myself (started several years ago!), but I've found it _absolutely crucial_ in working in ML, as it implicitly informs every single decision you make when designing, deploying, and scaling neural networks. Without it, I feel the field turns into an empirical "stabby stab-around in the dark" kind of game, which very much has its dopamine highs and lows, but ala Sutton, does not scale very well in the long-run. ;P

jackblemming 3y ago
Please list a single decision you've made that was directly influenced by Shannon's paper, and no I do not mean something you post hoc rationalized.
- jdkoeck 3y ago
  Good challenge, I hope we will get a response.
  - tysam_and 3y ago
    Your wish hath been granted! <3 :))))
- tysam_and 3y ago
  Sure. Basically everything in https://github.com/tysam-code/hlb-CIFAR10 https://github.com/tysam-code/hlb-CIFAR10 was directly founded on concepts shared in the above paper, down to the coding, commenting, and layout styles (hence why I advocate so strongly for it as a requirement for ML. The empirical benefits are clear to me). Before I sat down and wrote my first line, I spent a very long time thinking about how to optimize the repo. Not just in terms of information flow during training, but how the code was laid out (minimize the expected value of deltas for changes from a superset of possible code changes), to even the explanatory comments (ratio of space vs mental effort to decode the repo for experienced vs inexperienced developers). I really want it to be a good exemplary model of a different, more scalable, and more efficient way of conducting small-scale (and potentially resource-constrained) research. To do that, you have to maximize information efficiency at every stage of the pipeline, including temporally (!!!!). It's not perfect, but I've used info theory as a strong guiding light for that repo. There's more to say here, but it's a long conversation about the expected utility of doing research a few different kinds of ways.

xchip 3y ago

Ambiguous title, it should be "memory requirements for training LLMs"

d4rkp4ttern 3y ago
Thanks. This is why I read HN comments before the article :)

Havoc 3y ago

How feasible is it to train a toy model / very niche model on high end consumer hardware. Ie 24gb gpu? From scratch not tuning

sciolist 3y ago
It depends on the number of parameters you're going for - a billion parameters is going to be out of reach from scratch, but you could train some in the millions.

andirk 3y ago

These headlines read like owning a Tamagotchi virtual pet. Will it become popular to have ones own LLM, name it, and battle others!?