The results never fell off significantly with more training. Same model with longer training time on those bigger clusters should outdo it significantly. And

The results never fell off significantly with more training. Same model with longer training time on those bigger clusters should outdo it significantly. And they can expand the MoE model sizes without the same memory and bandwidth constraints. Still very surprising with so much less compute they were still able to do so well in the model architecture/hyperparameter exploration phase compared with Meta.