4 ms·

With mixture of experts (MoE) models, I would think that Mistral might make a lot of money reselling specific models to big players like Microsoft and Google to

With mixture of experts (MoE) models, I would think that Mistral might make a lot of money reselling specific models to big players like Microsoft and Google to augment their MoE systems. I mostly use either mistral7b or mixtral8-7b for most things, and experiment with other models on the side. In what world will LLMs not become a commodity? You might answer that question by saying that on player achieves AGI and captures the market, but I think there will be more to AGI than LLMs.

comeonbro 3y ago
> With mixture of experts (MoE) models, I would think that Mistral might make a lot of money reselling specific models to big players like Microsoft and Google to augment their MoE systems I am curious about what you mean by this. There is a (very understandable) common misconception about what "mixture of experts" means in current practice. Mixtral is not a mixture of fully-separable domain experts, like one is an expert at programming and one is an expert at arts and literature. The "experts" are per-layer (and as far as I know, the subject of their "expertise" not even interpretable at this point).