5 ms·
They didn't say LLM's don't scale. They said current LLM's don't scale, which appears to be true. LLM reasoning ability appears to be plateauing pretty hard, no
by wavemode 3y ago
They didn't say LLM's don't scale. They said current LLM's don't scale, which appears to be true. LLM reasoning ability appears to be plateauing pretty hard, no matter how much we spend.
- pama 3y agoScaling in this broader context can have many meanings. All of them disagree with the original statement and with yours. If you assume that OpenAI spent more on GPT3.5 than on GPT3, and more on GPT4 than on GPT3.5, which is certainly plausible, how do I understand your statement in that view? What is even meant with scaling stopping at GPT4 when the next model by OpenAI is not yet public? Of course it makes no sense to compare against yet another company throwing money at a problem, but within each company scaling to more money (training/modalities/tunings) as a function of time has improved reasoning abilities. I am probably misunderstanding something simple so any citation would help.
- deadbabe 3y agoI’m glad things are plateauing because when I suggested this could happen I was written off as an idiot. People just kept pointing to the leap from GPT 2 to GPT 3 and GPT 3 to GPT 4 and extrapolating that clearly progress will be exponential forever, some of those people were on this very site.
- famouswaffles 3y agoThere's pretty much nothing to suggest things are plateauing lol
- xadhominemx 3y agoThat GPT, Gemini, and Claude are all about the level of GPT-4 is evidence that things are plateauing -- not conclusive evidence, but evidence nonetheless.
- famouswaffles 3y agoNot Really. When GPT-3 was released, it would be well over a year before top competitors/research institutions with the compute hit >= GPT-3 levels. When you're in uncharted territory, you hit the levels you know are possible first and then you try to scale up. You waste less money this way. And scaling is only getting more expensive. Within the year, we can probably expect the release of Gemini 2, Claude 4, GPT-5 etc. If those are still small incremental improvements then we can talk about a slow down. Right now, it just seems par the course.
- xadhominemx 3y agoIn all the fast moving/low barrier technologies I've tracked over the years, competitors tend to leap frog each other. In this case, it's not happening despite scaled compute resources. Again, not conclusive but consistent with a slowdown.
- famouswaffles 3y agoYou've completely ignored what I just said. Competitors can leapfrog each other but it's not happening instantly. Gathering personnel, performing the necessary research, training, post-training red teaming. Each of these things take months of time and millions of dollars. That competitors are reaching GPT-4 level a year from it's release is perfectly normal and indicates nothing. Google released the paper for Gopher a year and a half after GPT-3 and that was never intended to be a product. >In all the fast moving/low barrier GPT-4 level compute is not low barrier. That's literally the main point of this paper.
- xadhominemx 3y agoIf 5+ companies can build it up in ~6 months, it's a low barrier
- famouswaffles 3y agoThere are only 2 models on GPT-4's level. 5+ companies? Lol
- deadbabe 3y agoNo one can seem to get past GPT 4 levels of performance. It’s done.
- famouswaffles 3y agoWhen do you think the first GPT-3 level model was trained post the release of GPT-3 ?
- globalise83 3y agoThe visible difference between answers that are broadly plausible and reliably exactly correct is going to be relatively small, probably only distinguishable by experts in a field. But the economic value and technological progress in achieving that leap will be colossal.
- ben_w 3y agoThis isn't necessarily just about the models themselves, as there also laws — and protestors dropping banners on Altman's head[0] — basically saying "can you not". [0] https://youtu.be/NjpNG0CJRMM?si=4iLD06yCbbSSfzm9&t=1013 https://youtu.be/NjpNG0CJRMM?si=4iLD06yCbbSSfzm9&t=1013