I work in this field. Derivatives (and in fact second derivatives or a Hessian matrix), when they are reliable and when you have them, makes your search faster.

I work in this field. Derivatives (and in fact second derivatives or a Hessian matrix), when they are reliable and when you have them, makes your search faster. Unfortunately for many problems, they are either unavailable or expensive to compute. (Imagine a case when your “function” is a full CFD simulation or a 2 hr job — every time an optimizer iterates it has to run an entire simulation. You can’t see the function and you can’t really calculate the derivatives — except crudely through finite differences). Derivative free optimizers choose to forgo derivative information and instead try to efficiently search the space to optimize the objective. Their convergence is often much slower than derivative based optimizers but they are also much simpler to implement and tend to not get stuck in valleys. All they really are are a way to cleverly evaluate f(X) to find the optimum, without needing f’(X) or f’’(X). Caveat: derivative free optimizers are terrible for constrained optimization however (especially if your constraints are anything but simple bounds). You can use barrier methods but still they’re not great. Also there’s an element of luck involved. It’s hard to benchmark derivative free methods because the methods themselves don’t matter too much (Nelder Mead Simplex method outperforms Powell’s method in some cases but not others. Powell’s method is well known but isn’t categorically better than any other algorithm) because when you don’t use derivatives, other higher order effects like initial point and shape of terrain dominate over the the choice of algorithm. I wouldn’t be too hung up on picking the right algorithm — instead I would focus my efforts on heuristics to get to a good initial point cheaply.