3 ms·
Why is the evidence on the side of the hype? Why do you assume something is size X just because nobody has proved it's smaller yet? The evidence is just whatev
by t43562 5mo ago
Why is the evidence on the side of the hype? Why do you assume something is size X just because nobody has proved it's smaller yet?
The evidence is just whatever it is - we cannot make predictions with it.
- aspenmartin 5mo agoWe have very stable trends in observed performance in a LOT of different measurements and stable scaling laws that continue to hold true across at the largest scales we have built. These scaling laws have a reasonable theoretical justification and make predictions which continue to hold true. Pretraining perplexity does seem related to downstream measures of performance but those too show very stable trends in performance gains as well, even if you look at these in isolation. Look at the epoch capability index as a good summary statistic across a number of benchmarks. So: yes we can and do make predictions with it and that’s how we get funding internally and externally to build at these scales. Bottlenecks arise about 3 orders of magnitude from now. On the other hand, any particular justification you have in mind for your point?
- t43562 5mo ago> So: yes we can and do make predictions with it and that’s how we get funding internally and externally to build at these scales