This whole '8 models not 1' thing is really confusing to me. My understanding of the MoE paper is that it's 8 FFNN modules attached to a single embedding and a

This whole '8 models not 1' thing is really confusing to me. My understanding of the MoE paper is that it's 8 FFNN modules attached to a single embedding and attention module. In that sense it is a 1tn+ parameter model, but a subset of those FFNN parameters are trained on each token/used in inference. OpenAI made a lot of noise about the power of scaling with raw compute, so I get why people are confused about why they are now in optimization mode, but the amount of disinformation I see about MoE right now is astounding.