Hi! I've worked a lot with ML algorithms, and I believe this is tied in with regret. Not only do huge steps (after an initial warmup) get to a rough minimum fas

Hi! I've worked a lot with ML algorithms, and I believe this is tied in with regret. Not only do huge steps (after an initial warmup) get to a rough minimum faster, due to the energy involved, the minimum found tends to be more shallow, because the parameters jitter around so much during training. It sort of makes sense too as you're approaching a finite goal in an optimization problem -- first, tiny steps to take any initial steps that might be huge due to any initialization differences, then annealing up to huge steps, then small steps again as you approach the end of your optimization process, and your signal:noise ratio is much lower.