3 ms·
Interesting, but it does not seem to be an overview of gradient optimisers, but rather gradient optimisers in ML, as I see no mentions of BFGS and the likes.
by sega_sai 2y ago
Interesting, but it does not seem to be an overview of gradient optimisers, but rather gradient optimisers in ML, as I see no mentions of BFGS and the likes.
- mike-the-mikado 2y agoI think the big difference is dimensionality. If the dimensionality is low, then taking account of the 2nd derivatives becomes practical and worthwhile.
- juliangoldsmith 2y agoWhat is it that makes higher order derivatives less useful at high dimensionality? Is it related to the Curse of Dimensionality, or maybe something like exploding gradients at higher orders?
- mike-the-mikado 2y agoIn n dimensions, the first derivative is an n-element vector. The second derivative is an n x n (symmetric) matrix. As n grows, the computation required to estimate the matrix increases (as at least n^2) and computation needed to use it increases (possibly faster). In practice, clever optimisation algorithms that use the 2nd derivative won't actually form this matrix.
- VHRanger 2y agoI'm also curious about gradient-less algorithms For non deep learning applications, Nelder-Mead saved my butt a fees times
- amelius 2y agoChatGPT also advised me to use NM a couple of times, which was neat.
- imurray 2y agoNelder–Mead has often not worked well for me in moderate to high dimensions. I'd recommend trying Powell's method if you want to quickly converge to a local optimum. If you're using scipy's wrappers, it's easy to swap between the two: https://docs.scipy.org/doc/scipy/reference/optimize.html#local-multivariate-optimization https://docs.scipy.org/doc/scipy/reference/optimize.html#loc... For nastier optimization problems there are lots of other options, including evolutionary algorithms and Bayesian optimization: https://facebookresearch.github.io/nevergrad/ https://facebookresearch.github.io/nevergrad/ https://github.com/facebook/Ax https://github.com/facebook/Ax
- woadwarrior01 2y agoLook into zeroth-order optimizers and CMA-ES.
- analog31 2y agoIt's with the utmost humility that I confess to falling back on "just use Nelder-Mead" in *scipy.optimize* when something is ill behaved. I consider it to be a sign that I'm doing something wrong, but I certainly respect its use.
- kernc 2y agoSAMBO does a good job of finding the global optimum in a black-box manner even compared to Nelder-Mead, according to its own benchmark ... https://sambo-optimization.github.io https://sambo-optimization.github.io
- deleted 2y ago[deleted]