What were the optimization algorithms you had most success with? Were they more successful in the sense of better out-of-sample error rate or in the sense of qu

What were the optimization algorithms you had most success with? Were they more successful in the sense of better out-of-sample error rate or in the sense of quicker convergence (or something else)?