3 ms·
As far as I can tell, this is a relevant proof when searching through high-dimensional parameter spaces (e.g. machine learning). This would mean that overall g
by tovej 6y ago
As far as I can tell, this is a relevant proof when searching through high-dimensional parameter spaces (e.g. machine learning).
This would mean that overall geometry is not a (big) factor when it comes to getting stuck on local optima. Depending on how a random walk is implemented however, the conditions might create a practically concave search space.
- scythmic_waves 6y ago> This would mean that overall geometry is not a (big) factor when it comes to getting stuck on local optima. I thought the results only applied to convex shapes. Search spaces in ML need not be convex, right? Or am I missing something?
- tovej 6y agoI assume they are convex, typically cuboid. This is the type of space you get when each parameter is searched for a certain range. Surely there could be search spaces that aren't convex. In that case the range of a variable would depend on the values of other variables. If you have an example of such a case I'd be interested in knowing about it.
- Bootvis 6y agoIf you have linear constraints the search space will be convex.
- scythmic_waves 6y agoAh I was confusing the loss landscape with the parameter search space because you said "local optima". Yep, I imagine most parameter search spaces are convex.
- 6gvONxR4sf7o 6y agoWhen it comes to getting stuck at a local optimum, I think it’s the convexity of the loss function that matters, not just the convexity of the parameter space. As I understand it, this result says that for convex losses, some simpler samplers work near enough to ideally.