4 ms·
I’m glad things are plateauing because when I suggested this could happen I was written off as an idiot. People just kept pointing to the leap from GPT 2 to GPT
by deadbabe 3y ago
I’m glad things are plateauing because when I suggested this could happen I was written off as an idiot. People just kept pointing to the leap from GPT 2 to GPT 3 and GPT 3 to GPT 4 and extrapolating that clearly progress will be exponential forever, some of those people were on this very site.
- famouswaffles 3y agoThere's pretty much nothing to suggest things are plateauing lol
- xadhominemx 3y agoThat GPT, Gemini, and Claude are all about the level of GPT-4 is evidence that things are plateauing -- not conclusive evidence, but evidence nonetheless.
- famouswaffles 3y agoNot Really. When GPT-3 was released, it would be well over a year before top competitors/research institutions with the compute hit >= GPT-3 levels. When you're in uncharted territory, you hit the levels you know are possible first and then you try to scale up. You waste less money this way. And scaling is only getting more expensive. Within the year, we can probably expect the release of Gemini 2, Claude 4, GPT-5 etc. If those are still small incremental improvements then we can talk about a slow down. Right now, it just seems par the course.
- xadhominemx 3y agoIn all the fast moving/low barrier technologies I've tracked over the years, competitors tend to leap frog each other. In this case, it's not happening despite scaled compute resources. Again, not conclusive but consistent with a slowdown.
- famouswaffles 3y agoYou've completely ignored what I just said. Competitors can leapfrog each other but it's not happening instantly. Gathering personnel, performing the necessary research, training, post-training red teaming. Each of these things take months of time and millions of dollars. That competitors are reaching GPT-4 level a year from it's release is perfectly normal and indicates nothing. Google released the paper for Gopher a year and a half after GPT-3 and that was never intended to be a product. >In all the fast moving/low barrier GPT-4 level compute is not low barrier. That's literally the main point of this paper.
- xadhominemx 3y agoIf 5+ companies can build it up in ~6 months, it's a low barrier
- famouswaffles 3y agoThere are only 2 models on GPT-4's level. 5+ companies? Lol
- xadhominemx 3y ago5+ companies with GPU clusters capable of training GPT-4 class models (as you said compute was the barrier)
- famouswaffles 3y agoWhat are the 5 companies? If most of them fall under, "the biggest and most valuable companies on the planet" then no, that's not low barrier.
- xadhominemx 3y agoGoogle Amazon Microsoft Meta Nvidia Oracle... if 6 companies can accomplish something in 6 months, that's low barrier. Expensive, but that's not the same as high barrier.
- deadbabe 3y agoNo one can seem to get past GPT 4 levels of performance. It’s done.
- famouswaffles 3y agoWhen do you think the first GPT-3 level model was trained post the release of GPT-3 ?
- globalise83 3y agoThe visible difference between answers that are broadly plausible and reliably exactly correct is going to be relatively small, probably only distinguishable by experts in a field. But the economic value and technological progress in achieving that leap will be colossal.