Isn't it amazing? One of those reminders that we don't understand DL as much as we like to pretend we do. Cyclic learning rates work well elsewhere, but I can'

Isn't it amazing? One of those reminders that we don't understand DL as much as we like to pretend we do. Cyclic learning rates work well elsewhere, but I can't think of any other case where switching made such a difference. I was completely shocked to read Chinchilla, and I'm still a little baffled - a cosine schedule? Really? That's it? You guys didn't make any other changes?