Interesting. But I wonder what simply scaling the model will do if we use the same kind of transformer model and the same kind of data (we might even run out of

Interesting. But I wonder what simply scaling the model will do if we use the same kind of transformer model and the same kind of data (we might even run out of internet data). Of course, we will probably see non-scaling innovation at least on the scale of GPT-3 to 4, and we could train on some AI-generated data.