4 ms·
LLM scaling laws are pretty well established at this point. They probably won’t hold forever but we aren’t at the breaking point yet. Some more pressing questi
by dwaltrip 3y ago
LLM scaling laws are pretty well established at this point. They probably won’t hold forever but we aren’t at the breaking point yet.
Some more pressing questions are:
* What new capabilities emerge as the models get better and better at predicting (i.e. loss goes down)?
* How much will it cost to train increasingly large models? And to run inference on them?
* How difficult will it be to find or generate more and more high quality data?
- gardenhedge 3y ago> LLM scaling laws are pretty well established at this point what are they then? I thought everyone was firmly in the "let's train with more data and see what happens" camp
- esafak 3y agohttps://en.wikipedia.org/wiki/Neural_scaling_law#Chinchilla_scaling_(Hoffmann,_et_al,_2022) https://en.wikipedia.org/wiki/Neural_scaling_law#Chinchilla_...
- dwaltrip 3y agohttps://openai.com/research/scaling-laws-for-neural-language-models https://openai.com/research/scaling-laws-for-neural-language... https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361
- kilotaras 3y agoScaling laws in terms of loss are well established. How loss translates into higher-level capabilities is anyone's guess.