4 ms·
> Small models only trained on 800B tokens, compared to 1T for llama-7B LLaMA is trained far beyond chinchilla optimality, so this is not as surprising to me.
by whimsicalism 4y ago
> Small models only trained on 800B tokens, compared to 1T for llama-7B
LLaMA is trained far beyond chinchilla optimality, so this is not as surprising to me.
- anentropic 4y agoAccording to this LLaMA still didn't go far enough: https://www.harmdevries.com/post/model-size-vs-compute-overhead/ https://www.harmdevries.com/post/model-size-vs-compute-overh...
- whimsicalism 4y agoYep, it depends on what your goal is.
- cubefox 4y agoThis doesn't say that LLaMA didn't go far enough.
- anentropic 4y agoNot exactly, but it did say they could have gone further than they did without wasting time and energy on infinitesimally small gains though
- dragonwriter 4y agoBut Chinchilla optimality, while an interesting result, is a strange target for most practical purposes. Training is one time, inference is many times; not training past the point where its cheaper to training a larger model for the same (proxy for) quality discounts to zero the import of the cost of inference.
- whimsicalism 4y agoYep, but if stability has the goal of training the best possible model then that would explain the choices they made.
- GaggiX 4y agoI mean 800B tokens on a 3B model and 7B model is still way beyond the Chinchilla scale.
- MacsHeadroom 4y agoThey're going to 1.5T and possibly 3T. The 800B is just for the "Alpha" checkpoints released today. New checkpoints will be released later.