Nelder Mead is sometimes effective, and it is pretty to visualize, but it gets stuck in some trivial cases. It can fail to find the bottom of a low dimensional

Nelder Mead is sometimes effective, and it is pretty to visualize, but it gets stuck in some trivial cases. It can fail to find the bottom of a low dimensional parabola, for instance. CMA-ES is worth looking at, and has a similar idea but a more robust algorithm. It's basically a cloud of points with some momentum: https://en.wikipedia.org/wiki/CMA-ES https://en.wikipedia.org/wiki/CMA-ES