4 ms·

Hi there! I'm one of the authors of the work. I can assure you the numbers are not incorrect: random search did not achieve good results in the semantic segment

Hi there! I'm one of the authors of the work. I can assure you the numbers are not incorrect: random search did not achieve good results in the semantic segmentation experiment. We had similar skepticism with reviewers at ICLR initially, and our new experiments convinced them. Random search is a good alternative and it did produce good results for the car counting experiment. We do not claim that RL is the best way to solve the problem at any point in our work. We just observe that random search did not perform well in the segmentation experiment while RL did perform well in all of our experiments. We think the RL formulation is a good one since it is flexible (can be easily adapted to neural network policies with thousands of weights for example). Hope this helped!

lalaland1125 7y ago
Hi Nataniel, Thanks for the response and paper. My main question here is that it doesn't seem to make sense that random search did significantly worse than random parameters? It seems that random search should only consistently lose in cases when the dev set performance is anti-correlated with the test set performance. Why do you think you saw random parameters beating random search? (Is there a typo in the table or something)?
- natanielruiz 7y ago
  No problem! I'm here to help clarify any doubts. I believe in this specific case, there was a local optimum that random search (with the parameters we selected, which we actually tuned as well) was not able to escape. In most cases I think you would find random search doing better than random parameters (unless we have the paradoxical situation you described), so your intuition is correct in the general case. But in this case there is no typo on the table! Does this answer your question?
  - lalaland1125 7y ago
    How could you have a local optimum with random search? It's not doing any hill climbing or anything so it shouldn't care at all about local optima. Yes, of course you could get bad results if you specify the random search distributions with too strong of a prior pointing in the wrong direction, but we usually specify a weak prior (uniform/etc distributions) for doing random search. What random distribution did you use for your random search? Anyways, if you had a bad random search prior, then that should have equally negatively impacted your random parameters as both should have been drawn from the same distribution. If you used a different random distribution for random search vs your "random parameters", why did you use a different distribution and why were they so different?
    - natanielruiz 7y ago
      Maybe we're talking about different algorithms. Do you agree that we are talking about random search as outlined in this Wikipedia page (https://en.wikipedia.org/wiki/Random_search https://en.wikipedia.org/wiki/Random_search)?
      - lalaland1125 7y ago
        Ah, that makes sense. By random search, I thought you were referring to random search as defined in http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a..... I believe that's the more common method for hyperparameter optimization cases like this (https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf https://static.googleusercontent.com/media/research.google.c..., https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html https://scikit-learn.org/stable/modules/generated/sklearn.mo...). It has much better guarantees than the method you used and tends to work quite well in practice. It's also probably what the reviewers meant by a random search baseline.
        natanielruiz 7y ago
        Our "random parameters" baseline is actually the random search that you refer to, while the "random search" that we use, is the actual derivative-free optimization technique of random search. Reviewers asked for a comparison with this specific random search. You can go look at the reviews on OpenReview.
        natanielruiz 7y ago
        I’m also confused when you say that it has better guarantees. Which guarantees?

lalaland1125 7y ago

Hi Nataniel, Thanks for the response and paper. My main question here is that it doesn't seem to make sense that random search did significantly worse than random parameters? It seems that random search should only consistently lose in cases when the dev set performance is anti-correlated with the test set performance. Why do you think you saw random parameters beating random search? (Is there a typo in the table or something)?

natanielruiz 7y ago
No problem! I'm here to help clarify any doubts. I believe in this specific case, there was a local optimum that random search (with the parameters we selected, which we actually tuned as well) was not able to escape. In most cases I think you would find random search doing better than random parameters (unless we have the paradoxical situation you described), so your intuition is correct in the general case. But in this case there is no typo on the table! Does this answer your question?
- lalaland1125 7y ago
  How could you have a local optimum with random search? It's not doing any hill climbing or anything so it shouldn't care at all about local optima. Yes, of course you could get bad results if you specify the random search distributions with too strong of a prior pointing in the wrong direction, but we usually specify a weak prior (uniform/etc distributions) for doing random search. What random distribution did you use for your random search? Anyways, if you had a bad random search prior, then that should have equally negatively impacted your random parameters as both should have been drawn from the same distribution. If you used a different random distribution for random search vs your "random parameters", why did you use a different distribution and why were they so different?
  - natanielruiz 7y ago
    Maybe we're talking about different algorithms. Do you agree that we are talking about random search as outlined in this Wikipedia page (https://en.wikipedia.org/wiki/Random_search https://en.wikipedia.org/wiki/Random_search)?
    - lalaland1125 7y ago
      Ah, that makes sense. By random search, I thought you were referring to random search as defined in http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a..... I believe that's the more common method for hyperparameter optimization cases like this (https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf https://static.googleusercontent.com/media/research.google.c..., https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html https://scikit-learn.org/stable/modules/generated/sklearn.mo...). It has much better guarantees than the method you used and tends to work quite well in practice. It's also probably what the reviewers meant by a random search baseline.
      - natanielruiz 7y ago
        Our "random parameters" baseline is actually the random search that you refer to, while the "random search" that we use, is the actual derivative-free optimization technique of random search. Reviewers asked for a comparison with this specific random search. You can go look at the reviews on OpenReview.
      - natanielruiz 7y ago
        I’m also confused when you say that it has better guarantees. Which guarantees?