3 ms·

How could you have a local optimum with random search? It's not doing any hill climbing or anything so it shouldn't care at all about local optima. Yes, of cour

How could you have a local optimum with random search? It's not doing any hill climbing or anything so it shouldn't care at all about local optima. Yes, of course you could get bad results if you specify the random search distributions with too strong of a prior pointing in the wrong direction, but we usually specify a weak prior (uniform/etc distributions) for doing random search. What random distribution did you use for your random search? Anyways, if you had a bad random search prior, then that should have equally negatively impacted your random parameters as both should have been drawn from the same distribution. If you used a different random distribution for random search vs your "random parameters", why did you use a different distribution and why were they so different?

natanielruiz 7y ago
Maybe we're talking about different algorithms. Do you agree that we are talking about random search as outlined in this Wikipedia page (https://en.wikipedia.org/wiki/Random_search https://en.wikipedia.org/wiki/Random_search)?
- lalaland1125 7y ago
  Ah, that makes sense. By random search, I thought you were referring to random search as defined in http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a..... I believe that's the more common method for hyperparameter optimization cases like this (https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf https://static.googleusercontent.com/media/research.google.c..., https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html https://scikit-learn.org/stable/modules/generated/sklearn.mo...). It has much better guarantees than the method you used and tends to work quite well in practice. It's also probably what the reviewers meant by a random search baseline.
  - natanielruiz 7y ago
    Our "random parameters" baseline is actually the random search that you refer to, while the "random search" that we use, is the actual derivative-free optimization technique of random search. Reviewers asked for a comparison with this specific random search. You can go look at the reviews on OpenReview.
  - natanielruiz 7y ago
    I’m also confused when you say that it has better guarantees. Which guarantees?

natanielruiz 7y ago

Maybe we're talking about different algorithms. Do you agree that we are talking about random search as outlined in this Wikipedia page (https://en.wikipedia.org/wiki/Random_search https://en.wikipedia.org/wiki/Random_search)?

lalaland1125 7y ago
Ah, that makes sense. By random search, I thought you were referring to random search as defined in http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a..... I believe that's the more common method for hyperparameter optimization cases like this (https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf https://static.googleusercontent.com/media/research.google.c..., https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html https://scikit-learn.org/stable/modules/generated/sklearn.mo...). It has much better guarantees than the method you used and tends to work quite well in practice. It's also probably what the reviewers meant by a random search baseline.
- natanielruiz 7y ago
  Our "random parameters" baseline is actually the random search that you refer to, while the "random search" that we use, is the actual derivative-free optimization technique of random search. Reviewers asked for a comparison with this specific random search. You can go look at the reviews on OpenReview.
- natanielruiz 7y ago
  I’m also confused when you say that it has better guarantees. Which guarantees?