I think there will be a trend where model's size will shrink due to better optimization / compression while hardware specs keep increasing. You can already see

I think there will be a trend where model's size will shrink due to better optimization / compression while hardware specs keep increasing. You can already see this with Chinchilla: https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b-greatly-outperforms-gpt-3-175b-and-gopher-280b-408b9b4510 https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b...