3 ms·
Isn't the consensus that the MOE architecture and other optimizations in the newest gen models (GPT-5, Gemini 3.0 to come, etc) will reduce inference costs by 5
by onlyrealcuzzo 1y ago
Isn't the consensus that the MOE architecture and other optimizations in the newest gen models (GPT-5, Gemini 3.0 to come, etc) will reduce inference costs by 50-75% already?
- yomismoaqui 1y agoSounds interesting, do you have some links with more info about this? Thanks!
- ACCount37 1y agoKind of. Frontier LLMs aren't going to get cheaper, but that's because the frontier keeps advancing. Price-performance though? The trend is clear: a given level of LLM capability keeps getting cheaper, and that trend is expected to hold. Improvements in architecture and training make LLMs more capability-dense, and advanced techniques make inference cheaper.
- onlyrealcuzzo 1y ago> Frontier LLMs aren't going to get cheaper One of the main selling points to MOE, is that the architecture is designed such that you can re-train experts independently, as well as add new experts, change the size of an experts parameters, etc, without retraining the entire model. If 80% of you usage comes from 20% of your experts, you can cut your future training costs SUBSTANTIALLY.
- ACCount37 1y agoI don't think anyone has ever managed to pull off a success story with this kind of advanced expert manipulation. It's not entirely impossible, but I remain skeptical until see a proof that it, first, works. And, second, that it actually has an advantage over "we'll just train another base model from scratch, but 10% larger, with those +5% performance architecture tweaks, and a new modality blender, and more of that good highly curated data in the dataset, and fresher data overall, and it'll be glorious".