3 ms·
I think it is interesting that llama 3 still does not use Mixture of Experts. Command R+ is also not using MoE. On paper you get 4x less training and inference
by DalasNoin 3y ago
I think it is interesting that llama 3 still does not use Mixture of Experts. Command R+ is also not using MoE. On paper you get 4x less training and inference compute cost. But maybe there is a good reason not to use it, this kind of supports this. Might be wasteful to create these huge MoE models if they can't outperform smaller dense models.
- samus 3y agoMany MoE models like Mixtral were not created from scratch. Somebody could still upcycle that 8B into an 8x8B. It should ideally be done by Meta since they have access to the original training data.
- CuriouslyC 3y agoMoE models are harder to fine tune, and they don't solve the biggest problem which is GPU memory use.