I think its almost certainly using at least two experts per token. It helps a lot during training to have two experts to contrast when putting losses on the exp

I think its almost certainly using at least two experts per token. It helps a lot during training to have two experts to contrast when putting losses on the expert router.