> Still, it shows that you can't just keep scaling the models and expect magic. They undertrained it, and the real benchmarks will be in the paper, so it's not

> Still, it shows that you can't just keep scaling the models and expect magic. They undertrained it, and the real benchmarks will be in the paper, so it's not at all obvious that this is not exactly what you would predict from existing scaling laws on GPT models (remember, they are power laws / log curves, so 3x doesn't move the needle that much even if it had been trained optimally). Plus of course, the history of these sorts of models like GPT-3 or CLIP is that the benchmarks are uninformative and you only get an idea of what improvements have emerged once the users get their hands on it and are able to discover the new induced capabilities. Nvidia/MS sometimes release their models, but mostly don't, so who knows... Anyway, this is a redundant submission to https://news.ycombinator.com/item?id=28827957 https://news.ycombinator.com/item?id=28827957