tldr; - 'sparse mixture of experts' means each layer contains 8 mini-neural-nets (experts), each trained to be good at certain data/tasks - a router network p

tldr; - 'sparse mixture of experts' means each layer contains 8 mini-neural-nets (experts), each trained to be good at certain data/tasks - a router network passes each token (word) to 2 experts which suit it best - since only 2/8 experts are utilized, the model effectively only uses 13/47B parameters during inference (text generation) - this expert mechanism makes it very efficient and effective (uses less params, is able to speciailize to tokens) - beats llmana 70b and gpt-3.5, especially at math and coding and language - has fine tuned model able to beat gemini pro, as well as llmana 70b and gpt-3.5