5 ms·

You could also try to devise optimal hyperparameters by genetic algorithm (or similar global optimizer) for every architecture with a big fixed number of genera

You could also try to devise optimal hyperparameters by genetic algorithm (or similar global optimizer) for every architecture with a big fixed number of generations. That should be apples to apples comparison... Or best to best. At the same time fragility of hyperparameters can be evaluated. I really appreciate the detailed analysis in the article.

abhgh 8y ago
Today you can do much better - there is a whole active and evolving sub-field of solving hyperparameter tuning as an optimization problem. An ex is performing bayesian optimization using gaussian processes. I hope the research from the community becomes mainstream soon.
- srean 8y ago
  One could optimize test error over hyperparameters by various methods. But at that point does test or validation error continue to provide the functionality that a test or validation set is supposed to provide ?
  - nonbel 8y ago
    Do you really consider this an obstacle? Obviously you need to have a validation set for the optimization and a new/unused test set for estimating out of sample performance.
  - abhgh 8y ago
    You are not doing away with the test/validation set. You are doing away with: 1. grid search - you would now only test for parameters that look promising (based on some criteria). Instead of covering the whole grid, you want to be smart about which points to try out. [1] 2. you might still do grid search but you want to preferentially allocate resources to parameter exploration based on how promising they seem. Hyperband - [2]- is an example. 3. you can do both - you can be smart about picking parameters, but for the ones you pick you can preferentially allocate resources. Ex *Bayesian Optimization and Hyperband* (BOHB) [1] Bayesian Optimization using Gaussian Processes is an example of this. https://arxiv.org/pdf/1206.2944.pdf https://arxiv.org/pdf/1206.2944.pdf Here's a library that helps you do this: https://github.com/JasperSnoek/spearmint https://github.com/JasperSnoek/spearmint. But there are other techniques in this family like Randomized Online Aggressive Racing (ROAR), Deep Network for Global Optimization (DNGO), Tree-Structured Parzen Estimators (TPE), etc [2] Hyperband: https://arxiv.org/abs/1603.06560 https://arxiv.org/abs/1603.06560 [3] BOHB: https://arxiv.org/abs/1807.01774 https://arxiv.org/abs/1807.01774
    - srean 8y ago
      Yeah that makes perfect sense -- optimizing hyperparameters more efficiently but on the train set.
bitL 8y ago
Genetic algorithms, Bayesian optimization, simulated annealing etc. are like 101 optimization algorithms, they won't get you anywhere. Hyperparameter tuning is as NP-hard as it gets in the same theoretical ballpark as cryptology when it comes to measuring how demanding it is. In Deep Learning you are basically doing meta-optimization, as the process of learning neural network itself is already non-linear optimization (hint: Adam is a non-linear optimizer); non-linear optimization is generally NP-hard unless restricted to some trivial cases like quadratic programming; here you want to optimize over an infinite set of already NP-hard problems.
- nonbel 8y ago
  What are you saying goes beyond "101 optimization algorithms"? Are you saying to put another "layer" of deep learning to tune the parameters of the first "layer"? Doesn't this just add even more parameters to tune?
  - bitL 8y ago
    No, you don't put another layer of Deep Learning (well, you technically could if you had some training data for optimization, but almost certainly you don't). You simply put another layer of (mixed-integer) non-linear optimization on top of Deep Learning hyperparameters, i.e. the hyper-parameters like learning rate, batch size, category weights, even the loss function composition, will be the variables you optimize. The (mixed-integer) non-linear optimization method will have its own set of parameters (they all do), so later you might indeed want to optimize those as well, if you get hold of some hyper-computing device allowing you to compress the first two steps to less than your lifetime. You can probably do this kind of hyper^n optimization as deeply as you like but I doubt it would lead you anywhere, given how terrible non-linear optimization performance on general functions is and how short the existence of Universe is.
    - nonbel 8y ago
      Sorry, its still unclear. Do you consider genetic algorithms, etc to be an example of (mixed-integer) non-linear optimization? Or are you talking about some other method? >"You can probably do this kind of hyper^n optimization as deeply as you like but I doubt it would lead you anywhere, given how terrible non-linear optimization performance on general functions is and how short the existence of Universe is." In practice you don't need to find the actual optimum though, just do better than the alternatives. This reminds me of "a single nn layer can approximate any function". Sure, but in practice a reasonable approximation will arise much easier with other architectures. EDIT: By "101" I thought you meant something like "low level" or "first thing you learn/try". Did you mean something else?
nmca 8y ago
"Population Based Training" goes some way to sorting this out, if you have a fuckton of compute available.

abhgh 8y ago

Today you can do much better - there is a whole active and evolving sub-field of solving hyperparameter tuning as an optimization problem. An ex is performing bayesian optimization using gaussian processes. I hope the research from the community becomes mainstream soon.

srean 8y ago
One could optimize test error over hyperparameters by various methods. But at that point does test or validation error continue to provide the functionality that a test or validation set is supposed to provide ?
- nonbel 8y ago
  Do you really consider this an obstacle? Obviously you need to have a validation set for the optimization and a new/unused test set for estimating out of sample performance.
- abhgh 8y ago
  You are not doing away with the test/validation set. You are doing away with: 1. grid search - you would now only test for parameters that look promising (based on some criteria). Instead of covering the whole grid, you want to be smart about which points to try out. [1] 2. you might still do grid search but you want to preferentially allocate resources to parameter exploration based on how promising they seem. Hyperband - [2]- is an example. 3. you can do both - you can be smart about picking parameters, but for the ones you pick you can preferentially allocate resources. Ex *Bayesian Optimization and Hyperband* (BOHB) [1] Bayesian Optimization using Gaussian Processes is an example of this. https://arxiv.org/pdf/1206.2944.pdf https://arxiv.org/pdf/1206.2944.pdf Here's a library that helps you do this: https://github.com/JasperSnoek/spearmint https://github.com/JasperSnoek/spearmint. But there are other techniques in this family like Randomized Online Aggressive Racing (ROAR), Deep Network for Global Optimization (DNGO), Tree-Structured Parzen Estimators (TPE), etc [2] Hyperband: https://arxiv.org/abs/1603.06560 https://arxiv.org/abs/1603.06560 [3] BOHB: https://arxiv.org/abs/1807.01774 https://arxiv.org/abs/1807.01774
  - srean 8y ago
    Yeah that makes perfect sense -- optimizing hyperparameters more efficiently but on the train set.

bitL 8y ago

Genetic algorithms, Bayesian optimization, simulated annealing etc. are like 101 optimization algorithms, they won't get you anywhere. Hyperparameter tuning is as NP-hard as it gets in the same theoretical ballpark as cryptology when it comes to measuring how demanding it is. In Deep Learning you are basically doing meta-optimization, as the process of learning neural network itself is already non-linear optimization (hint: Adam is a non-linear optimizer); non-linear optimization is generally NP-hard unless restricted to some trivial cases like quadratic programming; here you want to optimize over an infinite set of already NP-hard problems.

nonbel 8y ago
What are you saying goes beyond "101 optimization algorithms"? Are you saying to put another "layer" of deep learning to tune the parameters of the first "layer"? Doesn't this just add even more parameters to tune?
- bitL 8y ago
  No, you don't put another layer of Deep Learning (well, you technically could if you had some training data for optimization, but almost certainly you don't). You simply put another layer of (mixed-integer) non-linear optimization on top of Deep Learning hyperparameters, i.e. the hyper-parameters like learning rate, batch size, category weights, even the loss function composition, will be the variables you optimize. The (mixed-integer) non-linear optimization method will have its own set of parameters (they all do), so later you might indeed want to optimize those as well, if you get hold of some hyper-computing device allowing you to compress the first two steps to less than your lifetime. You can probably do this kind of hyper^n optimization as deeply as you like but I doubt it would lead you anywhere, given how terrible non-linear optimization performance on general functions is and how short the existence of Universe is.
  - nonbel 8y ago
    Sorry, its still unclear. Do you consider genetic algorithms, etc to be an example of (mixed-integer) non-linear optimization? Or are you talking about some other method? >"You can probably do this kind of hyper^n optimization as deeply as you like but I doubt it would lead you anywhere, given how terrible non-linear optimization performance on general functions is and how short the existence of Universe is." In practice you don't need to find the actual optimum though, just do better than the alternatives. This reminds me of "a single nn layer can approximate any function". Sure, but in practice a reasonable approximation will arise much easier with other architectures. EDIT: By "101" I thought you meant something like "low level" or "first thing you learn/try". Did you mean something else?

nmca 8y ago

"Population Based Training" goes some way to sorting this out, if you have a fuckton of compute available.