3 ms·
>It means you can train a chinchilla-optimal TinyLlama (1.1B param, 22B tokens) in 32 hours with 8 A100. They are training the model on 3000/22=136 times the v
by GaggiX 3y ago
>It means you can train a chinchilla-optimal TinyLlama (1.1B param, 22B tokens) in 32 hours with 8 A100.
They are training the model on 3000/22=136 times the value of the chinchilla scale. It will be interesting to see how much it will improve after way beyond this value.
- npsomaratna 3y agoPossibly a lot. See: https://espadrine.github.io/blog/posts/chinchilla-s-death.html https://espadrine.github.io/blog/posts/chinchilla-s-death.ht...
- isoprophlex 3y agoVery interesting, thanks for sharing!
- pluijzer 3y agoI now come to understand that the technobable in Star Trek wasn't that well predicted, in the future we will not be reversing polarities by alligning field cores. Picard will have us align our llamas with chiwawas to get an alpacafied chinchilla model.
- elpocko 3y agoLora and Alpaca at Tanagra.
- DarmokJalad1701 3y agoLlama, when the loss fell.
- kmlx 3y agofrom this episode if i’m not mistaken: https://en.m.wikipedia.org/wiki/Darmok https://en.m.wikipedia.org/wiki/Darmok i watched that series so many times…
- DarmokJalad1701 3y agoHence my username.
- koprulusector 3y agoThere’s should also be a tribble in there, somewhere.
- sp332 3y agoChinchilla predicts that you could get lower loss by training a larger model with that amount of data. But the model size in this case was chosen for other reasons, mostly speed of inference and cost of fine-tuning. So it's just irrelevant here.
- GaggiX 3y agoWell it's relevant if you want to compare the model trained optimally using the same amount of compute and this one parameter-bound to see how much you're trading.
- cypress66 3y agoIt's a bit amusing how people treat chinchilla scaling laws as a law of nature, when it's just about a certain architecture and dataset.