5 ms·
* Estimators for [Baeysian and Deep learning] approaches usually have to solve intractable optimization problems. Thus, they fall back on approximations and get
by YeGoblynQueenne 11y ago
* Estimators for [Baeysian and Deep learning] approaches usually have to solve intractable optimization problems. Thus, they fall back on approximations and get stuck in local maxima, and you don't really know what you're getting. *
So, with deep nets, how big a problem is getting stuck in local optima?
I mean, my intuition is that it's not magic, you're optimising a system of functions so you'd get stuck in local optima in the same way you get stuck with a single function. Is that generally the case?
- cinjon 11y agoIt seems that the local minima aren't actually a problem in these high dimensional spaces because most of them are either very close to the global minima or are saddle points, which can be escaped. There was a lot of work on this out of Bengio's lab. One such paper was Dauphin's "Identifying and attacking the saddle point..."
- YeGoblynQueenne 11y agoThanks, I'll see if I can find this paper.
- Teodolfo 11y agoI've never been stuck in a local optimum when training a big neural net. No one ever runs to convergence.