8 ms·

There is another reason why training deep neural networks is not as difficult as it sounds: the landscape of the loss function seems to be made of broad "U"-sha

There is another reason why training deep neural networks is not as difficult as it sounds: the landscape of the loss function seems to be made of broad "U"-shaped valleys that gently descend towards a small loss region. At initialization, the network is likely close to such valley, and once it gets there the rest of training is just a leisurely stroll. Formally, people have studied the spectrum of the Hessian and found that most of its eigenvectors are quite small with only a few, much larger ones. It all started with [1], with several recent extensions. [1] https://arxiv.org/pdf/1611.07476.pdf https://arxiv.org/pdf/1611.07476.pdf

aborsy 6y ago
This is just bunch of simulation, not a derivation.
- blackbear_ 6y ago
  True, but it has been confirmed experimentally several times. Unfortunately theoretical understanding in deep learning is quite lagging behind practical progress. This paper [1] seems to be getting closer to what you are looking for. [1] https://arxiv.org/abs/1910.02875 https://arxiv.org/abs/1910.02875
hammock 6y ago
Explain in layman's terms?
- macksd 6y ago
  Picture linear regression. If you have a bunch of data points you want to fit a line to, for any given line you can add up the vertical distance between all your data points and the line and come up with a measure of how inaccurate the line is. This is called your "loss". If you keep trying different values for the slope and intercept, you would find that this function is shaped like a big bowl, or valley. Regression is the process of repeatedly taking a step downhill until you can't go anywhere but up, and that must be the optimal line. Neural networks train in a similar way. You have a "loss" function that adds up how wrong your predictions are compared to the training data. You try different values for the weights in your neural network to see which ones send you downhill the fastest, step them in that direction, and repeat. Since the loss function in this case is more complex, it's not a single valley, but potentially many valleys. You can end up at a decent local minimum.
- pundesigner 6y ago
  Deep learning is a form of optimization. Optimization involves moving along a high-dimensional surface to find the lowest point. In principle this can be nearly impossible because the surface might be covered in dramatic peaks/valleys/saddles obscuring the route to the lowest point. Some simulations have implied that this is not what the surfaces corresponding to deep networks look like, and that they instead look like a big gentle slope down to the minimum, with only small bumps along the way.
  - asdfman123 6y ago
    Just like in life, the imagined obstacles are scarier than the real ones.
- sqrt17 6y ago
  In late 90s/early 2000s the mainstream thought around numerical optimization was that it was easy-ish when it was a linear problem, and if you had to rely on nonlinear optimization you were basically lost. People did EM (an earlier subgenre of what is now called Bayesian learning) but knew that it was sensitive to initialization and that they probably didn't hit a good enough maximum. Late 90s neural networks were basically a parlor trick - you could make it do little tricks but almost everything we have now including lots of compute, good initialization, regularization techniques, and pretraining, was absent in the late 90s. Then in the mid and later 2000s the mainstream method was convex optimization and you had a proof that there was one global optimum and a wide range of optimization methods were guaranteed to reach it from most initialization points. Simultaneously, the theory underlying SVMs and CRFs was developed - that you could actually do a large variety of things and still use these easy, dependable optimization techniques. And people hammered home the need for regularization techniques. In the late 2000s to early 2010s, several things again came together - one being the discovery of DropOut as a regularization technique - and the understanding that it was one, the other being the development of good initializers that made it possible to use deeper networks. Add to that improved compute power - including the development of CUDA which started out as a way to speed up texture computation but then led to general purpose GPU computing as we know it today. All this enabled a rediscovery of NN learning which could take off where linear learning methods (SVMs, CRFs) had plateaued before. And often you had a DNN that did what the linear classifier before did but could learn features in addition to that - and could be seen as finding a solution that was strictly better. But the lack of global optimum means that - even with good initializers and regularization packaged into the NN modules we use in modern DNN software implementations - the whole thing is way more finicky than CRFs ever were. (It would be wrong to say that CRFs are trivial to implement or never finicky at all, just as many well-understood NN architectures have a good out-of-the-box experience with TF/PyTorch etc. - so take this as a general statement that may not hold for all cases).
muppet_frog 6y ago
...hmm, that was counter to my understanding (limited though it may be...) which was partially formed by this paper: https://arxiv.org/abs/1712.09913 https://arxiv.org/abs/1712.09913 TLDR - loss landscapes are nasty, but you can tame them with skip connections.
- blackbear_ 6y ago
  These two papers are not necessarily contradicting each other, but perhaps my description was a bit sloppy. Sagun et al. (and derivative works) only focus on the Hessian on the trajectory followed by gradient descent, while Li et al. give a broader look at the loss surface as a whole.
UncleOxidant 6y ago
Doesn't the loss function landscape depend a lot on what you're trying to get the neural network to learn (what problem you're trying to solve)?
- taneq 6y ago
  Of course, but people seem to generally find it’s “egg-carton shaped” rather than “mountain range shaped”.
ivalm 6y ago
If anything for large models I thought the idea is everything is a saddle point. Your link looks at a relatively small dense network.
- mjburgess 6y ago
  My intuition is this: (1) decision surfaces are always linearly separable with enough dimensions (2) NNs have enough dimensions (3) NNs linear boundaries are coarse (4) coarse boundaries in high dimensions are likely to approximate the low-loss true boundary (ie., given 1). The idea behind (4) is just the linear regression idea: by (1) noise is gaussian and a straight-line is a good approximation. With a coarse line, we do not fit to noise, and hence prob. have a good aprox. The phrase "neural network" disguises the obviousness of this reasoning: a NN is just high-dimensional piece-wise linear regression. The only thing to be explained is why, in high dimensions, datasets end up nearly piece-wise linear. That isnt so hard to explain.
- ganzuul 6y ago
  As dimensionality increases the likelihood of a local minima decreases because there is almost always a dimension where the curvature is in the other direction.

aborsy 6y ago

This is just bunch of simulation, not a derivation.

blackbear_ 6y ago
True, but it has been confirmed experimentally several times. Unfortunately theoretical understanding in deep learning is quite lagging behind practical progress. This paper [1] seems to be getting closer to what you are looking for. [1] https://arxiv.org/abs/1910.02875 https://arxiv.org/abs/1910.02875

hammock 6y ago

Explain in layman's terms?

macksd 6y ago
Picture linear regression. If you have a bunch of data points you want to fit a line to, for any given line you can add up the vertical distance between all your data points and the line and come up with a measure of how inaccurate the line is. This is called your "loss". If you keep trying different values for the slope and intercept, you would find that this function is shaped like a big bowl, or valley. Regression is the process of repeatedly taking a step downhill until you can't go anywhere but up, and that must be the optimal line. Neural networks train in a similar way. You have a "loss" function that adds up how wrong your predictions are compared to the training data. You try different values for the weights in your neural network to see which ones send you downhill the fastest, step them in that direction, and repeat. Since the loss function in this case is more complex, it's not a single valley, but potentially many valleys. You can end up at a decent local minimum.
pundesigner 6y ago
Deep learning is a form of optimization. Optimization involves moving along a high-dimensional surface to find the lowest point. In principle this can be nearly impossible because the surface might be covered in dramatic peaks/valleys/saddles obscuring the route to the lowest point. Some simulations have implied that this is not what the surfaces corresponding to deep networks look like, and that they instead look like a big gentle slope down to the minimum, with only small bumps along the way.
- asdfman123 6y ago
  Just like in life, the imagined obstacles are scarier than the real ones.
sqrt17 6y ago
In late 90s/early 2000s the mainstream thought around numerical optimization was that it was easy-ish when it was a linear problem, and if you had to rely on nonlinear optimization you were basically lost. People did EM (an earlier subgenre of what is now called Bayesian learning) but knew that it was sensitive to initialization and that they probably didn't hit a good enough maximum. Late 90s neural networks were basically a parlor trick - you could make it do little tricks but almost everything we have now including lots of compute, good initialization, regularization techniques, and pretraining, was absent in the late 90s. Then in the mid and later 2000s the mainstream method was convex optimization and you had a proof that there was one global optimum and a wide range of optimization methods were guaranteed to reach it from most initialization points. Simultaneously, the theory underlying SVMs and CRFs was developed - that you could actually do a large variety of things and still use these easy, dependable optimization techniques. And people hammered home the need for regularization techniques. In the late 2000s to early 2010s, several things again came together - one being the discovery of DropOut as a regularization technique - and the understanding that it was one, the other being the development of good initializers that made it possible to use deeper networks. Add to that improved compute power - including the development of CUDA which started out as a way to speed up texture computation but then led to general purpose GPU computing as we know it today. All this enabled a rediscovery of NN learning which could take off where linear learning methods (SVMs, CRFs) had plateaued before. And often you had a DNN that did what the linear classifier before did but could learn features in addition to that - and could be seen as finding a solution that was strictly better. But the lack of global optimum means that - even with good initializers and regularization packaged into the NN modules we use in modern DNN software implementations - the whole thing is way more finicky than CRFs ever were. (It would be wrong to say that CRFs are trivial to implement or never finicky at all, just as many well-understood NN architectures have a good out-of-the-box experience with TF/PyTorch etc. - so take this as a general statement that may not hold for all cases).

muppet_frog 6y ago

...hmm, that was counter to my understanding (limited though it may be...) which was partially formed by this paper: https://arxiv.org/abs/1712.09913 https://arxiv.org/abs/1712.09913 TLDR - loss landscapes are nasty, but you can tame them with skip connections.

blackbear_ 6y ago
These two papers are not necessarily contradicting each other, but perhaps my description was a bit sloppy. Sagun et al. (and derivative works) only focus on the Hessian on the trajectory followed by gradient descent, while Li et al. give a broader look at the loss surface as a whole.

UncleOxidant 6y ago

Doesn't the loss function landscape depend a lot on what you're trying to get the neural network to learn (what problem you're trying to solve)?

taneq 6y ago
Of course, but people seem to generally find it’s “egg-carton shaped” rather than “mountain range shaped”.

ivalm 6y ago

If anything for large models I thought the idea is everything is a saddle point. Your link looks at a relatively small dense network.

mjburgess 6y ago
My intuition is this: (1) decision surfaces are always linearly separable with enough dimensions (2) NNs have enough dimensions (3) NNs linear boundaries are coarse (4) coarse boundaries in high dimensions are likely to approximate the low-loss true boundary (ie., given 1). The idea behind (4) is just the linear regression idea: by (1) noise is gaussian and a straight-line is a good approximation. With a coarse line, we do not fit to noise, and hence prob. have a good aprox. The phrase "neural network" disguises the obviousness of this reasoning: a NN is just high-dimensional piece-wise linear regression. The only thing to be explained is why, in high dimensions, datasets end up nearly piece-wise linear. That isnt so hard to explain.
ganzuul 6y ago
As dimensionality increases the likelihood of a local minima decreases because there is almost always a dimension where the curvature is in the other direction.