3 ms·
Does anyone have any numbers on how long it would take to train an N-billion parameter ChatGPT model on K-trillion tokens, for some reasonably large values of N
by 1024core 3y ago
Does anyone have any numbers on how long it would take to train an N-billion parameter ChatGPT model on K-trillion tokens, for some reasonably large values of N and K?
I'm guessing that at some stage, it would be hard(er) to keep these chips fed with training data. One would run into bandwidth issues keeping these monsters fed.
- eldenring 3y agoSince the cost per token of a forward/backward pass is so high on the models, the relative cost of a training set is small, and bandwidth (not talking about interconnect bandwidth) is even cheaper relatively.
- chaxor 3y agoYou can take a look at chinchilla or Llama scaling laws. It depends on what you want to optimize. If you're interested in inference optimal, go with llama, if it's compute optimal, go with chinchilla. GPT-4 technical paper also has good scaling laws for how benchmarks will behave given extrapolating from a small training set.