3 ms·
Skimmed the beginning of the paper, ISTM that a more accurate, less clickbait description of the work would be: For l-lipshitz-smooth convex functions, there is
by dooglius 3y ago
Skimmed the beginning of the paper, ISTM that a more accurate, less clickbait description of the work would be: For l-lipshitz-smooth convex functions, there is an existing best known bound on convergence rate with constant-step sizes. Paper proves a better bound with a variant gradient descent algorithm with occasional big steps. Unclear to me if the existing bound is known to be tight (i.e. if there exist known worst-case functions that are right at the bound).
Convex functions do not have suboptimal local optima, so this is seemingly unrelated to this as a known effective technique for escaping those, which other commenters are talking about.
- andy_ppp 3y agoAh, do you have a link to the paper?
- dooglius 3y agoThe "a study" in the article is a link https://arxiv.org/abs/2307.06324 https://arxiv.org/abs/2307.06324
- imtringued 3y agoSmooth convex functions? Kids these days. How about you talk about actually difficult optimization problems.
- thumbuddy 3y agoYea like the ones where we throw an entire AWS server room to try multiple random seeds, using gradient descent until it works, but honestly a RNG might work just as well, in a few contrived cases and we write our paper/make our product from it
- fnordpiglet 3y agoSeems a little dismissive. I think it’s a pretty interesting result from a theoretical point of view, and might be applied in some situations where you break a harder optimization problem into smaller problems with certain likely shapes of the piece wise problem. Regardless it’s not at all intuitive this pattern should have any optimality in any gradient descent space, so understanding it might yield some more fundamental concepts around more general optimization problems.