3 ms·
Anyone considering using this for continuous hyper-parameter optimization in machine learning should be warned that the automatic Lipschitz constant optimizatio
by alexbeloi 9y ago
Anyone considering using this for continuous hyper-parameter optimization in machine learning should be warned that the automatic Lipschitz constant optimization (the novelty of the method) will very likely not work well with a noisy objective function (any stochastic learning algorithm).
edit: Any grad students looking for a paper idea might consider extending this result to allow stochastic objective functions satisfying a stochastic Lipschitz condition.
- eesmith 9y agoQuoting from the linked-to page: > However, if you want to use LIPO in practice there are some issues that need to be addressed. The rest of this blog post discusses these issues and how the dlib implementation addresses them. First, if f(x) is noisy or discontinuous even a little it's not going to work reliably since k will be infinity. > ... Now each sample from f(x) has its own noise term, σi, which should be 0 most of the time unless xi is really close to a discontinuity or there is some stochasticity. Is that not what you were pointing out?