3 ms·
Scaling laws are largely about data and parameters, compute mostly more indirectly and weakly. If there's no extra data most scaling predictions start to look
by light_hue_1 2y ago
Scaling laws are largely about data and parameters, compute mostly more indirectly and weakly.
If there's no extra data most scaling predictions start to look bad.
None of this is surprising. Every AI/ML researcher has done a back of the envelope estimate to see how much data there is and when this must stop. Me included. It's a common topic at conferences.
- aix1 2y agoAm I understanding it correctly that the gist of your argument is that OpenAI, Google etc have run out of available data to train LLMs? What's your estimate for how many tokens that represents?