6 ms·
That makes sense and explains why it's unlikely to have local minimas, but I don't get the "no minimal at all" argument. Why no global minimas at all, just beca
by maxlamb 6y ago
That makes sense and explains why it's unlikely to have local minimas, but I don't get the "no minimal at all" argument. Why no global minimas at all, just because of high dimensionality?
- blackbear_ 6y ago> but I don't get the "no minimal at all" argument. That's just wrong, no need to think about it. Only unbounded loss functions don't have minima, but using such a loss function would not make sense.
- moultano 6y agoThis isn't true. A loss function that asymptotes can also have no minima, which commonly used loss functions do.
- bananaface 6y agoDoesn't the fact the network is discrete (floats have a maximum precision) mean this isn't actually the case? There's a finite number of states the net can be in, and one (or more) is best.
- dougabug 6y agoIt’s a heuristic argument that critical points are extremely unlikely to be local minima (ie positive definite second derivative). Loss surfaces of DNNs do typically have a global minimum (zero if they fit the training data exactly).
- PeterisP 6y agoArguably, a DNN seems likely to have many global minimums - given the level of (over)parametrization commonly used, a set of parameters that gets the lowest possible loss won't be unique, there will be huge sets of parameters that give exactly identical results.
- dougabug 6y agoDue to symmetry, at least, there are many global minima, but with the same minimum value.