There’s a few options. People think Gemini 1.5 is Sparse Mixture of Experts. (SMoE) Another One is self extend. https://arxiv.org/abs/2401.01325 https://arxiv

There’s a few options. People think Gemini 1.5 is Sparse Mixture of Experts. (SMoE) Another One is self extend. https://arxiv.org/abs/2401.01325 https://arxiv.org/abs/2401.01325 This paper also refers back to other options like yarn, etc.