3 ms·
I loved this post, thanks for writing it. I get the argument why one shouldn't expect local minima in very high dimensions. But then, what's wrong with the info
by Schiphol 6y ago
I loved this post, thanks for writing it. I get the argument why one shouldn't expect local minima in very high dimensions. But then, what's wrong with the informal argument that there has to be a minimum because, well, the expected loss cannot be negative?
- moultano 6y agoWith squared loss where it's easy for the loss to be zero, then yes, it will have lots of global minima, all at a loss of zero. For losses that asympotote, like logloss, they may have no minima.
- Schiphol 6y agoThanks. I guess my worry was that, once you are doing extremely well and your loss is very low, gradients are no longer independent, and will tend to go mostly up. Is this wrong?
- joshribakoff 6y agoI’m an ML newb but I think this would be true only of a converged model. Your model could always technically diverge in another epoch if learning rate is high enough and you process a batch of extreme outliers Even then, you may have still converged on a local optimum which was the take away I got from the article