3 ms·

Question about terminology: is superconvergence that particular schedule for LR/momentum updates, or is it a desirable emergent property that can be achieved by

Question about terminology: is superconvergence that particular schedule for LR/momentum updates, or is it a desirable emergent property that can be achieved by that LR/momentum schedule? I've been assuming it's the latter, but your usage sounds very much like the former. Is "1cycle" the schedule itself, or some other training policy?

cs702 8y ago
It's a desirable emergent property. AFAIK, no one knows of another way to achieve it other than by this kind of LR/momentum policy, so informally I think it's OK to use the term "superconvergence" (or "SuperC" for short) for the policy itself. "1 cycle" refers to having a "single cycle" over all epochs, as opposed to multiple cycles, for example as in SGDR. PS. Here's a good walkthrough of the training policy by a coauthor of the blogpost, with link to a sample notebook: https://sgugger.github.io/the-1cycle-policy.html https://sgugger.github.io/the-1cycle-policy.html