I reckon it would be useful if there were a set of open source MOE models with different sizes, like llama, for example 1.5B, 7B, 34B and 72B (or perhaps those

I reckon it would be useful if there were a set of open source MOE models with different sizes, like llama, for example 1.5B, 7B, 34B and 72B (or perhaps those numbers x the number of experts). It would enable much experimentation in the community. Not many are GPU rich enough to play with a 1.6T model!