4 ms·
A question for Jeremy, perhaps. For the longest time the fast.ai courses have used Adam and one-cycle, at least for CV tasks. More recently Ranger and flat-co
by fpgaminer 6y ago
A question for Jeremy, perhaps. For the longest time the fast.ai courses have used Adam and one-cycle, at least for CV tasks. More recently Ranger and flat-cos have been dominating the Imagenette leaderboards. I guess I'm curious if fast.ai intends to switch over to teaching that policy instead of one-cycle?
I guess more generally I'm curious what criteria the fast.ai team uses for deciding what techniques to teach. My feeling is that the courses have always taught the training techniques that are a healthy mix of SOTA, generally applicable, and easy to use.
Ranger + flat-cos has seemed like a really robust combo, and easy to use. So yeah, just interested in whatever internal discussions fast.ai may have had about it and other potential replacements for Adam + one-cycle.
- jph00 6y agoYes it's a great combo. I think fastai is the only library that actually has both of these built-in. However, because the LR warm-up is built-in to Ranger, it's actually a bit more fiddly to use - i.e. you really need to understand what it's doing. It doesn't work great with `Learner.fine_tune` and gradual unfreezing more generally, since you don't really want a full separate warmup for each phase. So I don't see it becoming the default or main optimizer shown in the course. But it's great to learn and use.