I would add Bayesian Optimization as a very strong member of family of derivative free algorithms. It assumes a black-box function, which also can be noisy (not

I would add Bayesian Optimization as a very strong member of family of derivative free algorithms. It assumes a black-box function, which also can be noisy (not all derivative-free algorithms can work well with noise). One of my favorite categories of algorithms. And its a fairly active area of research today, for applications in hyperparameter tuning.