3 ms·
If you are interested in this kind of thing, there are a whole host of "derivative free" optimization methods out there. One I've seen used in particular at a p
by TFortunato 6y ago
If you are interested in this kind of thing, there are a whole host of "derivative free" optimization methods out there. One I've seen used in particular at a previous job, the Nelder-Mead (downhill simplex) method, always struck me as pretty elegant and intuitive once you've seen it explained. https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method)
- TFortunato 6y agohttps://en.wikipedia.org/wiki/Coordinate_descent https://en.wikipedia.org/wiki/Coordinate_descent - Coordinate descent is another nice one that works well for certain problems.
- brummm 6y agoIn Python, scipy actually supports a lot of different optimization methods, including Nelder-Mead.
- lowdose 6y agoBut with scipy you have to offload to the cpu which is not always great. Tensorflow 1.0 had a complicated scipy loop integrated which caused that the scaling of gpu's was near impossible with tf.
- xscott 6y agoNelder Mead is sometimes effective, and it is pretty to visualize, but it gets stuck in some trivial cases. It can fail to find the bottom of a low dimensional parabola, for instance. CMA-ES is worth looking at, and has a similar idea but a more robust algorithm. It's basically a cloud of points with some momentum: https://en.wikipedia.org/wiki/CMA-ES https://en.wikipedia.org/wiki/CMA-ES
- twic 6y agoThere's a family of such methods invented by Michael Powell: https://en.wikipedia.org/wiki/PDFO https://en.wikipedia.org/wiki/PDFO From what i remember, they are "trust region" methods, which work by trying to fit a quadratic function to the local neighbourhood, then analytically finding the minimum of that. The clever stuff is in how you form and update the trust region efficiently.
- Enginerrrd 6y agoBoth Powell's and Nelder-mead are often the only thing I can get to work when the optimization is of a quantity derived from integrating a system of differential equations. Not sure why that is.
- abhgh 6y agoI would add Bayesian Optimization as a very strong member of family of derivative free algorithms. It assumes a black-box function, which also can be noisy (not all derivative-free algorithms can work well with noise). One of my favorite categories of algorithms. And its a fairly active area of research today, for applications in hyperparameter tuning.