I very much doubt we'll see continued 10x parameter growth per year for too many more years, although in the short term GPT-4 will confirm that trend (100T par

I very much doubt we'll see continued 10x parameter growth per year for too many more years, although in the short term GPT-4 will confirm that trend (100T params in a "few years"). Even if continued scaling of these models is deemed worthwhile, I doubt training hardware can keep up to keep the cost manageable. Even GPT-3's $6M training cost (& time) is significant. Does anyone have the stomach for $100M training costs, say? How many times are you willing to iterate that as you tweak the model? GPT-3 is a very interesting line of research though, given the non-intuitive capabilities as this type of model is scaled up. It'll certainly be interesting to see what GPT-4 is capable of. However, I doubt that these transformer architectures are capturing everything that the cortex is doing, and intelligence probably involves more than just the cortex anyway (hypothalamus, maybe cerebellum, etc). I expect in the next decade we'll see more research on elaborating these architectures, and making them more efficient, rather than just scaling them up much beyond the planned 100T GPT-4 type of size. Maybe by the time we've figured out the architecture for intelligence the hardware will have caught up to make training and runtime costs more practical.