DeepMind just published a new paper that dismantles the trend of building ever-larger language models, established by Kaplan and colleagues in 2020. They found

DeepMind just published a new paper that dismantles the trend of building ever-larger language models, established by Kaplan and colleagues in 2020. They found another factor that matters as much as model size: Data. The best models today are undertrained and oversized!