4 ms·

Only up to 64 variables? Why such small problems? And why are their results are averaged across all of the different test functions? I'd like to see the per

Only up to 64 variables? Why such small problems? And why are their results are averaged across all of the different test functions? I'd like to see the performance difference between Rosenbrock and Rastrigin, thank you very much. And they have a weird fixation on stopping rules, when in general your stopping rule is how many evaluations you can afford. Was this written by interns? There's no discussion of the retrogression by CMAES on small problems? What parameters does their algorithm have and how sensitive is it to its parameterization? What parameters did they use for the competing algorithms? If I were reviewing this paper I'd have these and other pointed questions for them. It's as if they did a cursory lit review and then decided to reinvent the entire field. Their one contribution seems to be not allowing you to run it on your own hardware.

indiesolver 8y ago
How long it will take CMAES to solve 10-dimensional Rosenbrock? Here, it takes about 700 function evaluations: https://indiesolver.com/tutorials/rosenbrock_python https://indiesolver.com/tutorials/rosenbrock_python Please note that their work is primarily for expensive optimization where you cannot afford millions of function evaluations like you would need on rotated Rastrigin to solve it exactly for n>64.
- sevensor 8y ago
  > How long it will take CMAES to solve 10-dimensional Rosenbrock? Here, it takes about 700 function evaluations It really depends on parameterization though. Did they use off-the-shelf parameters? Did they make any attempt to tune CMAES for their test functions? Did they make any attempt to tune their own optimizer for the test functions? It's really easy to put your thumb on the scale when you're writing a paper about a black-box optimizer, and my default reaction is to assume that any paper claiming to equal or outdo CMAES is pure B.S. This paper did nothing to convince me otherwise.
ScoutOrgo 8y ago
It is 64 hyper-parameters, not variables, which is a huge amount. Stopping rules are also a requirement of tuning. Can't just have a while True:. If an objective function doesn't improve after 100 (or any #), there needs to be logic to stop the trials since this system is apparently serving all of Alphabet.
- indiesolver 8y ago
  I think that in the context that he meant, 64 variables = 64 hyperparameters because it was about benchmark problems solved in black-box settings. In other words, it is not about the common misconception of comparing #hyperparameters and #variables on some ML model, where the latter can be in 10^7.
  - sevensor 8y ago
    You're correct -- I'm talking about how many variables control the behavior of the black box.
- sevensor 8y ago
  Surely that reinforces my point? That what matters isn't the stopping rule but your evaluation budget? If you can afford 100 evaluations, do 100 evaluations. Pick any stopping rule based on the objective function and I can construct you a scenario that either has you spending a long time on the wrong side of it, or stopping prematurely. Might as well have predictable behavior.
joshuamorton 8y ago
To expand on ScoutOrgo, >Only up to 64 variables? Why such small problems? This is a tool tuned for optimizing the hyperparameters of machine learning models. In practice, most models have, maybe, 10 hyperparameters in the normal sense (learning rate, nonlinearity, etc.). If you start talking about model structure as a hyperparameter, you can get vastly more than 64, but then these techniques aren't great anyway. So why such small problems? Because the blackbox function you're optimizing takes hours to run. So as another user mentions, if it takes you 700 function evaluations, you're gonna be running for the better part of a year. So the domain over which vizier works is one where you probably are never really using more than ~100 evaluations and often even then you would prefer to stop early if you meet certain conditions (because wasting significant compute doing unnecessary optimization is costly). As a concrete example, a single iteration of a relatively small and non-SOTA algorithm takes 20 minutes and $40 to run. [0] So stopping a day early saves you a day and $3000. Now scale that up 5 or 10x for larger models. (another way of putting this is that a day and $10K might be worth a .5% increase in accuracy, but isn't worth a .05% increase in accuracy. Stopping rules can encode that). [0]: http://www.fast.ai/2018/08/10/fastai-diu-imagenet/ http://www.fast.ai/2018/08/10/fastai-diu-imagenet/

indiesolver 8y ago

How long it will take CMAES to solve 10-dimensional Rosenbrock? Here, it takes about 700 function evaluations: https://indiesolver.com/tutorials/rosenbrock_python https://indiesolver.com/tutorials/rosenbrock_python Please note that their work is primarily for expensive optimization where you cannot afford millions of function evaluations like you would need on rotated Rastrigin to solve it exactly for n>64.

sevensor 8y ago
> How long it will take CMAES to solve 10-dimensional Rosenbrock? Here, it takes about 700 function evaluations It really depends on parameterization though. Did they use off-the-shelf parameters? Did they make any attempt to tune CMAES for their test functions? Did they make any attempt to tune their own optimizer for the test functions? It's really easy to put your thumb on the scale when you're writing a paper about a black-box optimizer, and my default reaction is to assume that any paper claiming to equal or outdo CMAES is pure B.S. This paper did nothing to convince me otherwise.

ScoutOrgo 8y ago

It is 64 hyper-parameters, not variables, which is a huge amount. Stopping rules are also a requirement of tuning. Can't just have a while True:. If an objective function doesn't improve after 100 (or any #), there needs to be logic to stop the trials since this system is apparently serving all of Alphabet.

indiesolver 8y ago
I think that in the context that he meant, 64 variables = 64 hyperparameters because it was about benchmark problems solved in black-box settings. In other words, it is not about the common misconception of comparing #hyperparameters and #variables on some ML model, where the latter can be in 10^7.
- sevensor 8y ago
  You're correct -- I'm talking about how many variables control the behavior of the black box.
sevensor 8y ago
Surely that reinforces my point? That what matters isn't the stopping rule but your evaluation budget? If you can afford 100 evaluations, do 100 evaluations. Pick any stopping rule based on the objective function and I can construct you a scenario that either has you spending a long time on the wrong side of it, or stopping prematurely. Might as well have predictable behavior.

joshuamorton 8y ago

To expand on ScoutOrgo, >Only up to 64 variables? Why such small problems? This is a tool tuned for optimizing the hyperparameters of machine learning models. In practice, most models have, maybe, 10 hyperparameters in the normal sense (learning rate, nonlinearity, etc.). If you start talking about model structure as a hyperparameter, you can get vastly more than 64, but then these techniques aren't great anyway. So why such small problems? Because the blackbox function you're optimizing takes hours to run. So as another user mentions, if it takes you 700 function evaluations, you're gonna be running for the better part of a year. So the domain over which vizier works is one where you probably are never really using more than ~100 evaluations and often even then you would prefer to stop early if you meet certain conditions (because wasting significant compute doing unnecessary optimization is costly). As a concrete example, a single iteration of a relatively small and non-SOTA algorithm takes 20 minutes and $40 to run. [0] So stopping a day early saves you a day and $3000. Now scale that up 5 or 10x for larger models. (another way of putting this is that a day and $10K might be worth a .5% increase in accuracy, but isn't worth a .05% increase in accuracy. Stopping rules can encode that). [0]: http://www.fast.ai/2018/08/10/fastai-diu-imagenet/ http://www.fast.ai/2018/08/10/fastai-diu-imagenet/