3 ms·

Cool, I'd be really interested in seeing that and might try out myself later. It'd be really interesting to see a comparison of the two on optimization problems

Cool, I'd be really interested in seeing that and might try out myself later. It'd be really interesting to see a comparison of the two on optimization problems with a large number of dimensions which stock CMA-ES takes a while (and a lot of samples) to get "warmed up".

chroem- 9y ago
I ran into some issues when comparing the two algorithms, but here is a comparison with CMA-ES. https://github.com/chrisstroemel/Simple/blob/master/cma-comparison.gif?raw=true https://github.com/chrisstroemel/Simple/blob/master/cma-comp... Simple is a constrained optimizer while CMA-ES is unconstrained, so I ended up choosing a sigma value such that the three sigma cutoff could be considered its faux "optimization domain." I gave the two algorithms equal areas to operate on, however CMA-ES is very sensitive to where it's initialized at. If I had started it near the global optimum it would have instantly converged without performing any optimization at all, so then there was also the issue of deciding on a "fair" starting location. Trying to get around this, I briefly tried the comparison on the Griewank function from -5:5, however CMA-ES showed nonconvergent behavior. CMA-ES seemed to have a higher startup cost than Simple, but very similar per-iteration runtime performance. Sample efficiency ranged from totally nonconvergent to instant convergence without global search, depending on where it was started. If you have any suggestions for a better testing methodology I would really appreciate them, because I'm still not quite satisfied with this comparison.
- captaindiego 9y ago
  Thanks that's quite interesting. Your results of seeing higher startup costs definitely mirror my experience. When I need constraints on CMA-ES for real world stuff I've been using a simple equation to map the output of the parameters before using them: y = a + (b - a)*(1 - cos(pi * x / 10)) / 2 Where a is min, b is max and x is the unconstrained input value. (I saw some random paper recommending this years ago and have been using it ever since). If I get a chance today I'll try and run your code and play around a bit. Really cool stuff!

chroem- 9y ago

I ran into some issues when comparing the two algorithms, but here is a comparison with CMA-ES. https://github.com/chrisstroemel/Simple/blob/master/cma-comparison.gif?raw=true https://github.com/chrisstroemel/Simple/blob/master/cma-comp... Simple is a constrained optimizer while CMA-ES is unconstrained, so I ended up choosing a sigma value such that the three sigma cutoff could be considered its faux "optimization domain." I gave the two algorithms equal areas to operate on, however CMA-ES is very sensitive to where it's initialized at. If I had started it near the global optimum it would have instantly converged without performing any optimization at all, so then there was also the issue of deciding on a "fair" starting location. Trying to get around this, I briefly tried the comparison on the Griewank function from -5:5, however CMA-ES showed nonconvergent behavior. CMA-ES seemed to have a higher startup cost than Simple, but very similar per-iteration runtime performance. Sample efficiency ranged from totally nonconvergent to instant convergence without global search, depending on where it was started. If you have any suggestions for a better testing methodology I would really appreciate them, because I'm still not quite satisfied with this comparison.

captaindiego 9y ago
Thanks that's quite interesting. Your results of seeing higher startup costs definitely mirror my experience. When I need constraints on CMA-ES for real world stuff I've been using a simple equation to map the output of the parameters before using them: y = a + (b - a)*(1 - cos(pi * x / 10)) / 2 Where a is min, b is max and x is the unconstrained input value. (I saw some random paper recommending this years ago and have been using it ever since). If I get a chance today I'll try and run your code and play around a bit. Really cool stuff!