If you have a smaller number of parameters, then quadratic descent (eg. Levenberg Marquardt) is a lot faster for most problems. Gradient descent is only used f

If you have a smaller number of parameters, then quadratic descent (eg. Levenberg Marquardt) is a lot faster for most problems. Gradient descent is only used for NN because we initialize all the parameters to random values, so the assumption of the JtJ approximating the Hessian (required by LM) doesn't hold, since residuals are too high. I suspect there might also be something about the learning rate effectively being too high when using quadratic descent. Even for large problems there are techniques like LBFGS which converge a lot faster than gradient descent, that don't require O(N^2) memory on the number of parameters.