3 ms·
Scaling Improvement has never been Linear though. Every next gen model so far has required at least an order of magnitude increase in compute, sometimes several
by famouswaffles 2y ago
Scaling Improvement has never been Linear though. Every next gen model so far has required at least an order of magnitude increase in compute, sometimes several more. So it's not a new revelation and these companies are aware of that. Microsoft for instance is building a 100B data center for a future next generation model releasing in 2028.
If models genuinely keep making similar leaps each generation then we're still a few generations before "More than a Microsoft".
- lispisok 2y agoSo at what point do the linear increases in capability not justify the exponential compute and data requirements, or when do we run out of resources to throw at it?
- famouswaffles 2y agoI never said I thought increase in ability was linear either. We're encroaching on phenomena that's genuinely hard to describe/put a number on but GPT-3 is worlds apart of 2 and it feels like 4 is easily ten times better than the OG 3. I can say Improvement lags behind compute somewhat but that’s really it. That said, it's ultimately up to the people footing the bill isn't it ?
- bredren 2y agoMicrosoft seems concerned enough to greatly increase its size: https://www.bloomberg.com/news/articles/2024-09-20/microsoft-s-ai-power-needs-prompt-revival-of-three-mile-island-nuclear-plant https://www.bloomberg.com/news/articles/2024-09-20/microsoft...