3 ms·
They seem to be adjusting lr between epochs as well when the loss explodes, not just runs. But I haven't read through the whole thing yet, maybe they trained th
by ackbar03 4y ago
They seem to be adjusting lr between epochs as well when the loss explodes, not just runs. But I haven't read through the whole thing yet, maybe they trained the whole thing properly from start to finish at the end. Otherwise that would be extremely hacky and irreproducible
- dotnet00 4y agoYeah I think for now they were just trying to get any comparable results due to a near complete lack of details on GPT-3. They seemed to have a hard deadline for the task.
- lumost 4y agoThe time and expense of training a model at this size does not benefit well from trial and error. It's simply impractical to iteratively try ~20 different learning schedules. Hideously ineficient and hacky to have someone manually tweaking things, but not terribly different from the state of the art for scientific research. As long as they state the objectives of their manual control and produce a log of what they did someone else could try to replicate it.