> The new model seems to prove that longstanding rumors of diminishing returns in training unsupervised-learning LLMs were correct and that the so-called "scali

> The new model seems to prove that longstanding rumors of diminishing returns in training unsupervised-learning LLMs were correct and that the so-called "scaling laws" cited by many for years have possibly met their natural end. The commonly-cited scaling laws[0] predict diminishing returns from scale. Fairly uncontroversial for instance that going from 1 GPU to 2 GPUs gives a larger improvement than going from 101 GPUs to 102 GPUs. Same with, say, computer graphics. I feel scaling laws are getting conflated with the theory of an exponential "hard takeoff" of self-improvement (which isn't particuarly well-founded in my opinion). [0]: https://arxiv.org/pdf/2001.08361 https://arxiv.org/pdf/2001.08361