4 ms·
Only 130 US homes’ annual power consumption to train GPT-3 is surprisingly small to me, considering it only needs to be done once.
by pxeger1 3y ago
Only 130 US homes’ annual power consumption to train GPT-3 is surprisingly small to me, considering it only needs to be done once.
- moffkalast 3y agoTbf, GPT-3 was made before the Chinchilla paper and was only trained on 0.3T tokens which is basically nothing for its size or for any current model (Mistral 7B, a 25x smaller model was trained on 8T). Doing it properly would require much more power.