4 ms·
> LLM scaling laws are pretty well established at this point what are they then? I thought everyone was firmly in the "let's train with more data and see what
by gardenhedge 3y ago
> LLM scaling laws are pretty well established at this point
what are they then? I thought everyone was firmly in the "let's train with more data and see what happens" camp
- esafak 3y agohttps://en.wikipedia.org/wiki/Neural_scaling_law#Chinchilla_scaling_(Hoffmann,_et_al,_2022) https://en.wikipedia.org/wiki/Neural_scaling_law#Chinchilla_...
- dwaltrip 3y agohttps://openai.com/research/scaling-laws-for-neural-language-models https://openai.com/research/scaling-laws-for-neural-language... https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361
- kilotaras 3y agoScaling laws in terms of loss are well established. How loss translates into higher-level capabilities is anyone's guess.