Leslie Smith has a number of great papers- this one is particularly relevant. I believe he coined the term “super-convergence” in the space of training neural n

Leslie Smith has a number of great papers- this one is particularly relevant. I believe he coined the term “super-convergence” in the space of training neural networks. Same idea- using very large learning rates post-warmup can speed up training an order of magnitude. https://arxiv.org/abs/1708.07120 https://arxiv.org/abs/1708.07120