If you've got an interesting nonlinear cost function to minimise and can compute the gradient, you can do a lot worse than plugging the thing into L-BFGS-B and

If you've got an interesting nonlinear cost function to minimise and can compute the gradient, you can do a lot worse than plugging the thing into L-BFGS-B and letting it optimise it. e.g. https://en.m.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80%93Goldfarb%E2%80%93Shanno_algorithm https://en.m.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%... https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_l_bfgs_b.html https://docs.scipy.org/doc/scipy/reference/generated/scipy.o... What'll that give you compared to a simple gradient descent? It'll attempt to accelerate convergence by estimating an approximation of the Hessian matrix - all the second order partial derivatives, and use a line search algorithm to figure out a good step size for each step instead of using an arbitrary constant step size aka "learning rate". The "L-" variation of the algorithm will use a limited amount of memory when approximating the Hessian matrix, which might help if your cost function has a larger number of parameters. The "-B" variations of the algorithm will also let you set upper and/or lower bounds on each variable that will be respected during the search.