3 ms·
You don't need to get good performance on everything with a single model—Mixture of Experts models can outperform equivalently sized monolithic models. My under
by lolinder 3y ago
You don't need to get good performance on everything with a single model—Mixture of Experts models can outperform equivalently sized monolithic models. My understanding is that GPT-4 is structured this way.
If we can trim a model in different ways to get different specializations, that could be really effective.
- novaRom 3y agoI just tried Mistral MoE 8x7B model and it works a bit faster than llama-2-70B but it looks it has almost the same skills. In fact, all latest models of 13B-70B size are quite similar. Could it be large part of their training data is the same?
- cyanydeez 3y agothere might be a laws of average thing going on? Central Limit Theorem? there's a finite number of relevant language tokens. the trick of current LLM is finding what's basically the center of a vast series of probability.