3 ms·

If I didn't know that neural nets worked, and someone explained the idea, I'd strongly expect them to get stuck in local minima, instead of learning this elabor

by colah3 11y ago

If I didn't know that neural nets worked, and someone explained the idea, I'd strongly expect them to get stuck in local minima, instead of learning this elaborate behavior. (That's basically what happens with a normal RNN, I think.) It's probably a case of my intuitions about optimization being really broken in high-dimensional spaces, but I'd like to improve that. I'd also like to understand how the cost surface we're optimizing on interacts with network architecture decisions. Both, unfortunately, are very hard demands.

hyperbovine 11y ago
Unless there is some major advance in nonconvex optimization here, (sub)gradient descent on the function you describe in your post is almost certainly converging to a local minimum as well. I guess maybe the surprise is that local minima can perform well? But again, this does not seem like such a surprise when you consider the number of parameters being fitted. I do not have much experience with LSTMs per se, but modern i.e. conv-nets are basically O(10^8)-dimensional nonlinear maps. It just doesn't seem all that unexpected that you can embed a huge number of patterns into such an object.
nabla9 11y ago
>If I didn't know that neural nets worked, and someone explained the idea, I'd strongly expect them to get stuck in local minima This has been the general consensus (based on intuition) until recently. It turns out that it's not as big problem as tought. Almost all local minima have very similar error values. The bigger issue is combinatorially large number of saddle points where gradient becomes zero with very few downward curving directions. Saddle-free Newton method is one way around that. Open Problem: The landscape of the loss surfaces of multilayer networks, Choromanska, LeCun, Arous http://jmlr.org/proceedings/papers/v40/Choromanska15.pdf http://jmlr.org/proceedings/papers/v40/Choromanska15.pdf

by colah3 11y ago

hyperbovine 11y ago

Unless there is some major advance in nonconvex optimization here, (sub)gradient descent on the function you describe in your post is almost certainly converging to a local minimum as well. I guess maybe the surprise is that local minima can perform well? But again, this does not seem like such a surprise when you consider the number of parameters being fitted. I do not have much experience with LSTMs per se, but modern i.e. conv-nets are basically O(10^8)-dimensional nonlinear maps. It just doesn't seem all that unexpected that you can embed a huge number of patterns into such an object.

nabla9 11y ago

>If I didn't know that neural nets worked, and someone explained the idea, I'd strongly expect them to get stuck in local minima This has been the general consensus (based on intuition) until recently. It turns out that it's not as big problem as tought. Almost all local minima have very similar error values. The bigger issue is combinatorially large number of saddle points where gradient becomes zero with very few downward curving directions. Saddle-free Newton method is one way around that. Open Problem: The landscape of the loss surfaces of multilayer networks, Choromanska, LeCun, Arous http://jmlr.org/proceedings/papers/v40/Choromanska15.pdf http://jmlr.org/proceedings/papers/v40/Choromanska15.pdf