4 ms·

Actually, quite a few algorithms can provably find global optima. However, there's a property of such algorithms that makes them (often, but not always) impract

Actually, quite a few algorithms can provably find global optima. However, there's a property of such algorithms that makes them (often, but not always) impractical for many problems. Any algorithm which can find a global optimum must necessarily sample its input space densely. That means that to be sure that we have the global optimum, we must evaluate the function within every epsilon-ball in our input space. If we didn't, then we could construct a function which was the same as some function we had found the optimum for everywhere except for in a single epsilon-ball, and which had a lower value than the minimum value in that ball. Then, our algorithm wouldn't find this new minimum and thus would not be a true global optimizer. This property means we can't really provably obtain global optima without prohibitive numbers of function evaluations. However, for most "normal" functions, these algorithms typically work quite well and are commonly used in derivative-free black-box optimization. One famous and easy-to-understand example is the DIRECT algorithm. The paper describing this algorithm is quite well-written and easy to read, and well worth your time if you're interested in global optimizers.

flir 3y ago
Which paper? Is it this? https://link.springer.com/article/10.1007/BF00941892 https://link.springer.com/article/10.1007/BF00941892 I'm willing to give it a shot but that abstract doesn't scream "easy to read" to me. What's an epsilon-ball? I'm guessing it somehow describes the resolution you need to sample at in order to guarantee you've found all the local maximums? I guess what I'm struggling with is that, with a probabilistic search through a space you don't already know the properties of, I don't see how that can be anything other than checking every solution in turn - in other words a brute force search. Does this all boil down to "the worse-case scenario of simulated annealing is brute force search"? And where does the logarithmic cooling schedule come in?
- archermarks 3y ago
  Yeah, looking at it again it's not as accessible as I remember. It's quite mathy, but well-written. An epsilon-ball is just a "ball" (circle or sphere analogue of dimension n, where n is the number of inputs) of radius epsilon (some arbitrarily small number). So you're correct that it relates to the resolution at which we can be said to "know" the solution. With simulated annealing, the cooling schedule dictates how and when the algorithm switches from exploration to exploitation. With a logarithmic cooling schedule, the cooling rate is 1 / log (1 + t), so the temperature approaches zero extremely slowly, and the algorithm basically never switches out of exploration mode. You're correct that this is basically a brute force search of the domain, and this is the implication of the theorem I mention above--ANY global optimizer reduces to brute force search in the worst case.
guyomes 3y ago
Another derivation-free optimization algorithm is the Cross-Entropy Method [1]. The main idea is to use a random generator to sample 100 points for example. Then take the 20 points with maximal evaluations. Finally adapt your random generator to maximize the chances of sampling those 20 points, and start again. When used for continuous functions, you can use Gaussian random generator, and this becomes similar to the CMA-ES algorithm [2]. [1]: https://en.wikipedia.org/wiki/Cross-entropy_method https://en.wikipedia.org/wiki/Cross-entropy_method [2]: https://en.wikipedia.org/wiki/CMA-ES https://en.wikipedia.org/wiki/CMA-ES
- archermarks 3y ago
  That's a neat method! I hadn't seen it before.

flir 3y ago

Which paper? Is it this? https://link.springer.com/article/10.1007/BF00941892 https://link.springer.com/article/10.1007/BF00941892 I'm willing to give it a shot but that abstract doesn't scream "easy to read" to me. What's an epsilon-ball? I'm guessing it somehow describes the resolution you need to sample at in order to guarantee you've found all the local maximums? I guess what I'm struggling with is that, with a probabilistic search through a space you don't already know the properties of, I don't see how that can be anything other than checking every solution in turn - in other words a brute force search. Does this all boil down to "the worse-case scenario of simulated annealing is brute force search"? And where does the logarithmic cooling schedule come in?

archermarks 3y ago
Yeah, looking at it again it's not as accessible as I remember. It's quite mathy, but well-written. An epsilon-ball is just a "ball" (circle or sphere analogue of dimension n, where n is the number of inputs) of radius epsilon (some arbitrarily small number). So you're correct that it relates to the resolution at which we can be said to "know" the solution. With simulated annealing, the cooling schedule dictates how and when the algorithm switches from exploration to exploitation. With a logarithmic cooling schedule, the cooling rate is 1 / log (1 + t), so the temperature approaches zero extremely slowly, and the algorithm basically never switches out of exploration mode. You're correct that this is basically a brute force search of the domain, and this is the implication of the theorem I mention above--ANY global optimizer reduces to brute force search in the worst case.

guyomes 3y ago

Another derivation-free optimization algorithm is the Cross-Entropy Method [1]. The main idea is to use a random generator to sample 100 points for example. Then take the 20 points with maximal evaluations. Finally adapt your random generator to maximize the chances of sampling those 20 points, and start again. When used for continuous functions, you can use Gaussian random generator, and this becomes similar to the CMA-ES algorithm [2]. [1]: https://en.wikipedia.org/wiki/Cross-entropy_method https://en.wikipedia.org/wiki/Cross-entropy_method [2]: https://en.wikipedia.org/wiki/CMA-ES https://en.wikipedia.org/wiki/CMA-ES

archermarks 3y ago
That's a neat method! I hadn't seen it before.