4 ms·

If you've ever used "Bayesian optimization" to choose hyperparameters, that was almost definitely a Gaussian process. The article focused on the case where you

If you've ever used "Bayesian optimization" to choose hyperparameters, that was almost definitely a Gaussian process. The article focused on the case where you have a finite number of test points, which is probably a good idea for an article like this. Still, there is another interpretation of Gaussian processes where they are actual stochastic processes (hence the name), a probability distribution over a set of functions. I would have found an article that covered that interpretation even more helpful, although I'm not sure an easy-to-follow version could exist.

Zephyr314 8y ago
While a lot of Bayesian optimization methods use GPs (MOE, Spearmint, BayesOpt, etc) some use TPEs as well (most notably, hyperopt [0]), and some ensemble these methods and others like SigOpt (YC W15) [1] (disclaimer: I'm one of the founders). I tried to briefly go over the functional interpretation of GPs in this talk [2], although the book by Rasmussen and Williams does a much more thorough job [3] (free online, check out chapter 2 for this approach). I'm happy to answer any questions about the differences. If you're a student/academic SigOpt is also completely free [4]. [0]: http://hyperopt.github.io/hyperopt/ http://hyperopt.github.io/hyperopt/ [1]: https://sigopt.com/research/ https://sigopt.com/research/ [2]: https://www.youtube.com/watch?v=J6UcAdH54RE&list=PLbSwfqjMfjGc6Z-WP9VYBCZsHHyOImabQ https://www.youtube.com/watch?v=J6UcAdH54RE&list=PLbSwfqjMfj... [3]: http://www.gaussianprocess.org/gpml/chapters/RW.pdf http://www.gaussianprocess.org/gpml/chapters/RW.pdf [4]: https://sigopt.com/solution/for-academia https://sigopt.com/solution/for-academia
- mlevental 8y ago
  this person has responded 3 times in this thread with the same links. aren't there rules against shilling?
  - sah2ed 8y ago
    The superfluous posts are now dead. Rules working as intended.
abhgh 8y ago
While this is mostly true there is has been some research to use non-GP models for Bayesian Optimization or the general Sequential Model Based Optimization (SMBO) problem. Some examples: * SMAC (using Random forests) - [1] * Using Deep Neural Networks - [2] * Tree Structured Parzen Estimators - [3] [1] https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf [2] https://arxiv.org/pdf/1502.05700.pdf https://arxiv.org/pdf/1502.05700.pdf [3] https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf https://papers.nips.cc/paper/4443-algorithms-for-hyper-param...

Zephyr314 8y ago

While a lot of Bayesian optimization methods use GPs (MOE, Spearmint, BayesOpt, etc) some use TPEs as well (most notably, hyperopt [0]), and some ensemble these methods and others like SigOpt (YC W15) [1] (disclaimer: I'm one of the founders). I tried to briefly go over the functional interpretation of GPs in this talk [2], although the book by Rasmussen and Williams does a much more thorough job [3] (free online, check out chapter 2 for this approach). I'm happy to answer any questions about the differences. If you're a student/academic SigOpt is also completely free [4]. [0]: http://hyperopt.github.io/hyperopt/ http://hyperopt.github.io/hyperopt/ [1]: https://sigopt.com/research/ https://sigopt.com/research/ [2]: https://www.youtube.com/watch?v=J6UcAdH54RE&list=PLbSwfqjMfjGc6Z-WP9VYBCZsHHyOImabQ https://www.youtube.com/watch?v=J6UcAdH54RE&list=PLbSwfqjMfj... [3]: http://www.gaussianprocess.org/gpml/chapters/RW.pdf http://www.gaussianprocess.org/gpml/chapters/RW.pdf [4]: https://sigopt.com/solution/for-academia https://sigopt.com/solution/for-academia

mlevental 8y ago
this person has responded 3 times in this thread with the same links. aren't there rules against shilling?
- sah2ed 8y ago
  The superfluous posts are now dead. Rules working as intended.

abhgh 8y ago

While this is mostly true there is has been some research to use non-GP models for Bayesian Optimization or the general Sequential Model Based Optimization (SMBO) problem. Some examples: * SMAC (using Random forests) - [1] * Using Deep Neural Networks - [2] * Tree Structured Parzen Estimators - [3] [1] https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf [2] https://arxiv.org/pdf/1502.05700.pdf https://arxiv.org/pdf/1502.05700.pdf [3] https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf https://papers.nips.cc/paper/4443-algorithms-for-hyper-param...