3 ms·
Other companies have done this (see Qwen Coder). It doesn't scale past a few disciplines like math and code though, and using mixtures of experts give you most
by KTibow 1y ago
Other companies have done this (see Qwen Coder). It doesn't scale past a few disciplines like math and code though, and using mixtures of experts give you most of the same benefits.
- littlestymaar 1y agoUnlike what their names imply, MoE don't have a concept of domain expert at all. It's just a fancy name for sparse evaluation of the total network to save compute and memory bandwidth.