3 ms·
The results of the experiment seem counterintuitive just because the used learning rates are huge (up to 10 or even 100). These are not the lr you would use in
by alexmolas 3y ago
The results of the experiment seem counterintuitive just because the used learning rates are huge (up to 10 or even 100). These are not the lr you would use in a normal setting. If you look at the region of small lr it seems all of them converge.
So I would say the experiment is interesting, but not representative of real world deep learning.
In the experiment, you have a function of 272 variables with a lot of minima and maxima, and at each gradient descent step you take huge steps (due to big lr). So my intuition is that convergence is more a matter of luck rather than hyperparameters.
- vintermann 3y agoYes, but as he also says, the best hyperparameters are near the boundary. You want to be a bit greedy here.
- alexmolas 3y agoThe way the author defines best hyperparameters is also a bit weird. To assign a "score" to each pair of HP the author sums all the losses during training (ie score = ∑ᵀₜ₌₀ lₜ) instead of just taking the value of the loss at the last epoch (score = l_T). This makes the converging solutions with high learning rates appear better since they have lower losses since the beginning (so the sum is smaller), but it doesn't mean that the final value of the loss is better than the ones with smaller learning rates.
- dmarchand90 3y agoI think the article is very honest about this just being a fun exploration. They even show how you can get similar patterns with newton's algorithm which is a more "classical" take
- alexmolas 3y agoYes, the author is clear in this regard. But I've seen people interpreting this paper as "training deep networks is chaotic", and I don't think that's the case. I interpret it more as "if you're not careful with your learning rate your training will be chaotic"
- repelsteeltje 3y agoYes. I immediately had visions of a misconfigured PID controller or similar chaotic emergence encountered in control theory. I was surprised and delighted though, that this type of chaos has so much fractal beauty.
- nyrikki 3y agoNote that not all fractals are chaotic. Inter dimensionality is the defining feature of fractals. A multilayer ANN will have compression that may be fractal like. But in theory a feed forward network could be single layer but will a lot more neurons. I have the feeling this result was due to representation but there are some features that look like riddled basins. Riddled basins do arise in SNN or spikey neutral networks that have continuous output bs the binary output of ANNs. But as all PAC learning is just compression that may be the cause also. I'll be digging into this on the weekend though.
- breather 3y agoWith all do respect, why are you commenting here rather than replying to the person giving your comment context?
- sdenton4 3y agoThere was a nice paper about five years ago that also found that the best hyperparams were in the bounty of divergence, even for 'real' imagenet models.
- locuscoeruleus 3y agoWhat paper was that?
- sdenton4 3y agoHere you go: https://arxiv.org/abs/1811.03600 https://arxiv.org/abs/1811.03600
- lopuhin 3y agoIf convergence were a matter of luck, it would look completely different, like white noise, but it clearly has well-defined structure. The reason for high learning rate is that they used full batched training (see the first cell in https://colab.research.google.com/github/Sohl-Dickstein/fractal/blob/main/the_boundary_of_neural_network_trainability_is_fractal.ipynb https://colab.research.google.com/github/Sohl-Dickstein/frac...), and when batch sizes are large, learning rates typically can be large as well. Plus as others said it's more of a toy problem, it would be hard to get such detail on anything non-toy.