I'm not in ML, but one thing I remember from school (a million years ago, before ML was big) is that Newton's method needs a matrix inversion every iteration, w

I'm not in ML, but one thing I remember from school (a million years ago, before ML was big) is that Newton's method needs a matrix inversion every iteration, which is expensive when you have a lot of decision variables. Not sure why other old school algorithms like conjugate gradient aren't used though. I guess for rectified linear activation functions the second derivative isn't useful. Maybe that's it.