They're saving money by the simple fact that sparse models are much less expensive to train and infere than dense equivalents. >OpenAI themselves have also pub

They're saving money by the simple fact that sparse models are much less expensive to train and infere than dense equivalents. >OpenAI themselves have also publicly acknowledged they're hitting diminishing returns on model size. [1] Do people even properly read this ? Altman acknowledged they were hitting economic wall scaling not that they thought they couldn't get better performance scaling further. Ilya believes there is plenty performance left to squeeze still. There was even an interview specifically correcting this misinformation.