2 ms·
Critical bit: > However, while these insights may change how researchers think about gradient descent, they likely won’t change how the technique is currently
by bell-cot 3y ago
Critical bit:
> However, while these insights may change how researchers think about gradient descent, they likely won’t change how the technique is currently used. Grimmer’s paper focused only on smooth functions, which have no sharp kinks, and convex functions, which are shaped like a bowl and only have one optimal value at the bottom. These kinds of functions are fundamental to theory but less relevant in practice; the optimization programs machine learning researchers use are usually much more complicated...
And machine learning is by no means the only area in which the "real world" curves subject to optimization have, ah, "problematic" shapes.