4 ms·

Turning to MoE now that we see they don't have to underperform dense counterparts has nothing to do with diminishing returns. It's simply more economically viab

Turning to MoE now that we see they don't have to underperform dense counterparts has nothing to do with diminishing returns. It's simply more economically viable

somenameforme 3y ago
With a multi expert system you end up making way more inferences per query, and depending on exactly how its built up, you may even end up requiring more training as well. There could be some extremely domain specific way they're saving some money, somehow, but it seems generally unlikely. It's all about diminishing returns. OpenAI themselves have also publicly acknowledged they're hitting diminishing returns on model size. [1] In any case this happens in literally every single domain that uses neural networks. You make this extremely rapid and unbelievably progress which leads one to start looking forward to where this is leading, and the literally infinite possibilities. But then at some point each single percent point improvement starts costing more and more. Initially a bit more compute can help you get over these hurdles, but then it becomes clear you're headed towards an asymptote that's far from where you want to be. It's why genuine fully self driving vehicles look much further away today than they did ~8 years ago. [1] - https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/ https://www.wired.com/story/openai-ceo-sam-altman-the-age-of...
- famouswaffles 3y ago
  They're saving money by the simple fact that sparse models are much less expensive to train and infere than dense equivalents. >OpenAI themselves have also publicly acknowledged they're hitting diminishing returns on model size. [1] Do people even properly read this ? Altman acknowledged they were hitting economic wall scaling not that they thought they couldn't get better performance scaling further. Ilya believes there is plenty performance left to squeeze still. There was even an interview specifically correcting this misinformation.

Turning to MoE now that we see they don't have to underperform dense counterparts has nothing to do with diminishing returns. It's simply more economically viable

somenameforme 3y ago

With a multi expert system you end up making way more inferences per query, and depending on exactly how its built up, you may even end up requiring more training as well. There could be some extremely domain specific way they're saving some money, somehow, but it seems generally unlikely. It's all about diminishing returns. OpenAI themselves have also publicly acknowledged they're hitting diminishing returns on model size. [1] In any case this happens in literally every single domain that uses neural networks. You make this extremely rapid and unbelievably progress which leads one to start looking forward to where this is leading, and the literally infinite possibilities. But then at some point each single percent point improvement starts costing more and more. Initially a bit more compute can help you get over these hurdles, but then it becomes clear you're headed towards an asymptote that's far from where you want to be. It's why genuine fully self driving vehicles look much further away today than they did ~8 years ago. [1] - https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/ https://www.wired.com/story/openai-ceo-sam-altman-the-age-of...

famouswaffles 3y ago
They're saving money by the simple fact that sparse models are much less expensive to train and infere than dense equivalents. >OpenAI themselves have also publicly acknowledged they're hitting diminishing returns on model size. [1] Do people even properly read this ? Altman acknowledged they were hitting economic wall scaling not that they thought they couldn't get better performance scaling further. Ilya believes there is plenty performance left to squeeze still. There was even an interview specifically correcting this misinformation.