Okay yeah totally, that makes sense. We also have stopped seeing huge gains in scaling beyond GPT3 sized models which would indicate we have hit some maximum th

Okay yeah totally, that makes sense. We also have stopped seeing huge gains in scaling beyond GPT3 sized models which would indicate we have hit some maximum there. Although we thought we had hit a maximum with deep learning before the transformer came around so it could be misleading. Distributing the computation to many models like what Google did with GLaM could very well be the future of AGI. Economies of models rather than one big model.