4 ms·
Does the time to train the model increase linearly with the number of parameters, or exponentially? In other words, GPT-3 is 17,500X the number of parameters b
by pcthrowaway 4y ago
Does the time to train the model increase linearly with the number of parameters, or exponentially?
In other words, GPT-3 is 17,500X the number of parameters but does that mean you can train it in 17,500X the amount of time it takes to train the 10M param model?
- tysam_and 4y agoI am not from the LLM world, but I believe it's mostly constrained by the standard multiprocessing limits -- communication and synchronization of multiple workers, some of whom operate over an exceedingly slow Ethernet interface.
- bitL 4y agoIn theory it should be linear, however, the parallelization is not perfect and some overlapping parts of gradients are computed on multiple GPUs at the same time so expect some constant factor slowdown on average.
- sebzim4500 4y agoOn top of what other people have said about parallelism overheads, you normally need more data to train a bigger network and the training time is roughly proportional to network size * training data. IIRC OpenAI used a million times more data to train GPT3 than karpathy used in this video, so a naive estimate would be that it would take about 20 billion times more compute. This is could be a significant overestimate since Karpathy probably used each bit of the training set more times than openAI did.