Jeremy from fast.ai here. Thanks for the thoughtful summary. A few more clarifications which hopefully are of some interest: - Sylvain tried AdamW + 1cycle on

Jeremy from fast.ai here. Thanks for the thoughtful summary. A few more clarifications which hopefully are of some interest: - Sylvain tried AdamW + 1cycle on many different datasets and with different architectures. The 5-10x improvement over regular SGD and LR annealing was not a rare occurrence, but was the most common result - Accuracy was always at least about as good as the regular approach, and usually better - So our main result here is to strongly suggest that AdamW + 1cycle should be the default for most neural net training - The goal of this research was to improve the fastai library, not to write a paper. But since the results were so practically useful we figured we'd take the time to document them in a blog post so others can benefit too - fastai is a self-funded (i.e out of my own pocket) non-profit research lab.