The size of neural nets grew 3000x in 10 years - from 60M params AlexNet to 175B params GPT-3. Thats about 2.23x per year. Moore's law was a doubling every two

The size of neural nets grew 3000x in 10 years - from 60M params AlexNet to 175B params GPT-3. Thats about 2.23x per year. Moore's law was a doubling every two years, that comes up at 2^(10/2)=32x. Model size scaled 100x faster than compute over a decade. We are paying for this difference by using more energy and hardware, but it's already too expensive to train except for a few labs, and deployment is restricted. Can't even load GPT-3 on most computers. Stable Diffusion is an exception, they did a good job and were lucky the model can be so small.