3 ms·
If true, does this imply the end of the scaling laws? I'm guessing all these players were expecting a certain level of model improvement for their compute budge
by aix1 2y ago
If true, does this imply the end of the scaling laws? I'm guessing all these players were expecting a certain level of model improvement for their compute budget. If they're now disappointed, this suggests their estimates were off?
Or is this a disconnect between pertaining loss and downstream performance on real-world tasks?
- iLoveOncall 2y agoIt was never a "law", it was what companies selling models made their investors believe but the limits were obvious from the start.
- pixelbash 2y agoI suspect it’s more the end of an avenue than the end of the road, but it sure seems like throwing more data and compute is hitting diminishing returns on existing methods.
- yzydserd 2y agoPunctuated equilibrium
- light_hue_1 2y agoScaling laws are largely about data and parameters, compute mostly more indirectly and weakly. If there's no extra data most scaling predictions start to look bad. None of this is surprising. Every AI/ML researcher has done a back of the envelope estimate to see how much data there is and when this must stop. Me included. It's a common topic at conferences.
- aix1 2y agoAm I understanding it correctly that the gist of your argument is that OpenAI, Google etc have run out of available data to train LLMs? What's your estimate for how many tokens that represents?
- ForHackernews 2y agoIs the obstacle compute, or is it available training data? At some point the LLMs are just redigesting their own vomit.
- jasfi 2y agoVery likely the training data.