I think with Mixtral Medium they mean MoE 2x13B which is on top on huggingface leaderboard? It is still not close to 8x175B, but size alone is not most importan

I think with Mixtral Medium they mean MoE 2x13B which is on top on huggingface leaderboard? It is still not close to 8x175B, but size alone is not most important factor. With smarter training methods and data it is possible we will see performance similar to gpt-4 in open source mixture of experts of smaller sizes.