4 ms·
A 5x to 10x reduction in training time versus other approaches is impressive, regardless of how it's achieved. Hours of waiting become minutes; days become hour
by cs702 8y ago
A 5x to 10x reduction in training time versus other approaches is impressive, regardless of how it's achieved. Hours of waiting become minutes; days become hours; weeks become days.
Among the examples mentioned:
* They fine-tuned a Resnet50 to 90% accuracy on the Cars Stanford Dataset in 60 epochs vs 600 in previous reports.
* They trained an AWD LSTM RNN from scratch to state-of-the-art perplexity on Wikitext-2 in 90 epochs vs 750 epochs in previous reports.
* The trained a QRNN from scratch to state-of-the-art perplexity on Wikitext-2 in 90 epochs vs 500 epochs in previous reports.
There are more examples of improvements in training speed (and accuracy) in the blog post, which provides persuasive evidence that a training regime combining (a) "AdamW" (which fixes a well-known issue with weight-decay in Adam) and (b) "superconvergence" (i.e., raising then decreasing LR, and doing the inverse with momentum) now appears to be the best/fastest way to train deep neural nets.
Interestingly, no one yet really knows why or how AdamW and SuperC together can achieve a 10x improvement in training efficiency.
- claytonjy 8y agoQuestion about terminology: is superconvergence that particular schedule for LR/momentum updates, or is it a desirable emergent property that can be achieved by that LR/momentum schedule? I've been assuming it's the latter, but your usage sounds very much like the former. Is "1cycle" the schedule itself, or some other training policy?
- cs702 8y agoIt's a desirable emergent property. AFAIK, no one knows of another way to achieve it other than by this kind of LR/momentum policy, so informally I think it's OK to use the term "superconvergence" (or "SuperC" for short) for the policy itself. "1 cycle" refers to having a "single cycle" over all epochs, as opposed to multiple cycles, for example as in SGDR. PS. Here's a good walkthrough of the training policy by a coauthor of the blogpost, with link to a sample notebook: https://sgugger.github.io/the-1cycle-policy.html https://sgugger.github.io/the-1cycle-policy.html
- jph00 8y agoJeremy from fast.ai here. Thanks for the thoughtful summary. A few more clarifications which hopefully are of some interest: - Sylvain tried AdamW + 1cycle on many different datasets and with different architectures. The 5-10x improvement over regular SGD and LR annealing was not a rare occurrence, but was the most common result - Accuracy was always at least about as good as the regular approach, and usually better - So our main result here is to strongly suggest that AdamW + 1cycle should be the default for most neural net training - The goal of this research was to improve the fastai library, not to write a paper. But since the results were so practically useful we figured we'd take the time to document them in a blog post so others can benefit too - fastai is a self-funded (i.e out of my own pocket) non-profit research lab.
- cs702 8y agoThank YOU. I posted this comment here only because the prior top comment was unfairly negative, in my view. Sometimes I'm afflicted by this: https://xkcd.com/386/ https://xkcd.com/386/ Had you posted on the main thread I would have upvoted your comment over mine. Have you considered accepting donations from third parties?
- bitL 8y agoHow can we send you $/BTC donations for funding support? Couldn't find it anywhere on fast.ai.