3 ms·
Is there a name for this theorem? I'd be interested in reading more about it.
by MereInterest 8y ago
Is there a name for this theorem? I'd be interested in reading more about it.
- asplake 8y agoIf I understand the Wikipedia page for SGD correctly, it’s the Robbins Siegmund theorem.
- vbarrielle 8y agoThat theorem only implies global convergence for convex or pseudoconvex functions. For a generic function it only guarantees convergence to a local minimum.
- _0ffh 8y agoMaybe GP is talking about the hypothesis that in high-dimensional space you will probably almost always find /some/ direction which decreases the error function. IIRC this hypothesis is well supported, but not yet proven. So the real problem may not be local minima but saddle points that just /appear/ to be minima due to slow optimisation progress. For more details, check out https://arxiv.org/abs/1406.2572 https://arxiv.org/abs/1406.2572
- deleted 8y ago[deleted]
- lawrenceyan 8y agoNo Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis [1]. Authored by Rong Ge, an absolute prodigy in the field if I might say. I'm sure we'll be seeing a great deal more from him in solidifying the theory behind the current day trial and error sort of deep learning we have today. [1] https://arxiv.org/abs/1704.00708 https://arxiv.org/abs/1704.00708
- selimthegrim 8y agoHe also did “On the optimization landscape of tensor decompositions” with Tengyu Ma which I thought was really cool.
- lawrenceyan 8y agoDefinitely, I second a read on this paper in addition to the first.