I think they probably did a lot of hyperparameter searching to train the smaller models and then extrapolated for the largest model, but I'm just guessing. Open

I think they probably did a lot of hyperparameter searching to train the smaller models and then extrapolated for the largest model, but I'm just guessing. OpenAI had a finite amount of money when they were training GPT-3, they likely do it differently now that inference costs are significant compared to training costs.