A good run-down of the different algorithms: https://ruder.io/optimizing-gradient-descent/ https://ruder.io/optimizing-gradient-descent/ I think people favour a

A good run-down of the different algorithms: https://ruder.io/optimizing-gradient-descent/ https://ruder.io/optimizing-gradient-descent/ I think people favour adaptive learning rate options like Adam in practice since they generally do seem to perform well, and are often less sensitive to initial conditions and the exact hyper-parameters used. There will always be people who like to test N optimizers with parameter sweeps to squeeze a tiny bit of extra performance out, but for the rest of us the default Adam or AdamW options and good, unobjectionable choices :)