3 ms·

Thanks. I guess my worry was that, once you are doing extremely well and your loss is very low, gradients are no longer independent, and will tend to go mostly

• joshribakoff 6y ago
  I’m an ML newb but I think this would be true only of a converged model. Your model could always technically diverge in another epoch if learning rate is high enough and you process a batch of extreme outliers Even then, you may have still converged on a local optimum which was the take away I got from the article