duvenaud

13 ms·

1. 121.▲
   by duvenaud 11y ago

   Right, once we start heavily hyper-parameterizing our models, there's the potential to overfit the hyperparameters. I think most people agree that has already happened on common datasets such as MNIST. However, the current method of a
2. 122.▲
   by duvenaud 11y ago

   Exactly. The problem is that it's hard to find a good balance between having too many hyperparameters, versus basically telling it the answer beforehand. In particular, it was hard to figure out how to let it learn the stride of the
3. 123.▲
   by duvenaud 11y ago

   Thanks! Yes, great suggestion. One could also use a similar trick to learn the number of layers, by regularizing unused layers to be close to the identity function. One thing that we wanted to try but didn't have time was to learn a
4. 124.▲
   by duvenaud 11y ago

   Yes, gradient-based hyperparameter optimization is one option for taking the human out of the loop to some extent. Of course, we can't avoid having to specify hyper-hyper-parameters! More generally, automatic hyperparameter tuning (wi
5. 125.▲
   by duvenaud 11y ago

   Authors here, feel free to ask us anything!