3 ms·
The Chinchilla paper was called "Training Compute-Optimal Large Language Models", which is exactly on point.
by cubefox 4y ago
The Chinchilla paper was called "Training Compute-Optimal Large Language Models", which is exactly on point.
- dragonwriter 4y agoI mean, it would be slightly more accurate if it had been “Compute-Optimal Training of Large Language Models”, since the models so-trained aren’t the thing that is compute-optimal, the training is compute-optimal. But… yeah, its hardly a title chosen for marketing rather than description.
- sroussey 4y agoBut we all know what you mean when you say “the Chinchilla paper”
- moritzwarhier 4y agoStupid question, is training considered a part of the model? Or is this only common parlance for GPT (the P).
- cubefox 4y agoInterestingly, the models themselves also have compute cost associated with them, namely the cost of running them (inference). The smaller a model is, the less compute it needs for inference. Which is an interesting trade-off, because you can overtrain (rather than undertrain, as in the past) a model, i.e. use more tokens, fewer parameters, and ultimately more compute than optimal during training, to get lower inference cost. https://www.harmdevries.com/post/model-size-vs-compute-overhead/ https://www.harmdevries.com/post/model-size-vs-compute-overh... This website has an interesting graph which visualizes this: > For example, the compute overhead for 75% of the optimal model size is only 2.8%, whereas for half of the optimal model size, the overhead rises to 20%. As we move towards smaller models, we observe an asymptotic trend, and at 25% of the compute-optimal model size, the compute overhead increases rapidly to 188%. So if you train your model you shouldn't just look at compute-optimal training but try to anticipate how much the model will probably be used for inference, to minimize total compute cost. Basically, the less inference you expect to do with your model, the closer the training should be to Chinchilla-optimality.
- ResearchCode 4y agoThat's better than other meme titles, but compute is still not a noun.
- cubefox 4y agoIs there a good replacement in noun form?
- ResearchCode 4y agocomputing power