Tbf, GPT-3 was made before the Chinchilla paper and was only trained on 0.3T tokens which is basically nothing for its size or for any current model (Mistral 7B

Tbf, GPT-3 was made before the Chinchilla paper and was only trained on 0.3T tokens which is basically nothing for its size or for any current model (Mistral 7B, a 25x smaller model was trained on 8T). Doing it properly would require much more power.