3 ms·
My understanding of this phenomenon in DL is that its not due to anything intrinsic to gradient descent, the same principles and understanding apply. Rather, i
by huevosabio 2y ago
My understanding of this phenomenon in DL is that its not due to anything intrinsic to gradient descent, the same principles and understanding apply.
Rather, it is that with very large dimensionality the probability that you spuriously get all derivatives to be zero is vanishingly small. That is, local minima are less likely because you need a lot of dimensions to agree that df(x_i)/dx_i = 0.
I may be wrong though!
- dxbydt 2y agoif the probability that you get a derivative close to 0 is small, say only 10%, then you need just 3 dimensions to get that multiplicative probability equal to a tenth of a percent. 3 is hardly “very large dimensionality” you can assign different numbers, but still you will find you don’t need more than say 10 dimensions for this effect to happen.