Behavior in high dimensions is even weirder than you might think, because you're nearly always close to a saddle point. There's way more saddle points than mini

Behavior in high dimensions is even weirder than you might think, because you're nearly always close to a saddle point. There's way more saddle points than minima or maxima (consider modeling the sign of eigenvalues for the Jacobian as a binomial distribution -- how likely is it they're all positive or all negative?). However usually only a relatively small subset of dimensions are important (read: represent a stable optimum), see the 'lottery ticket' hypothesis for more. What's also weird is that "convergence" in these cases isn't really convergence, it's just slowly floating between saddle points as you begin to overfit. Hence early stopping. > Gradient descent in high dimensions also seems to have the ability to "step over" areas of high loss This has much less to do with gradients per se and way more to do with step size. It's an artifact of discretization, not of the algorithm per se.