4 ms·

> We propose to instead learn the hyperparameters themselves by gradient descent, and furthermore to learn the hyper-hyperparameters by gradient descent as well

• oh_sigh 7y ago
  It's been thought of, but the standard view was that it was way too expensive to perform when humans are pretty good at fiddling with them to get good enough results after a little bit of practice.
  • PeterisP 7y ago
    Ah, the GDGS (gradient descent by grad student) approach where you estimate the gradient direction using an educated guess, tweak the system towards that, run an experiment, write down the results and repeat until you're happy (or run out of time).
• gwern 7y ago
  No, it's been done. For example, "Gradient-based Hyperparameter Optimization through Reversible Learning" https://arxiv.org/abs/1502.03492 https://arxiv.org/abs/1502.03492 , Maclaurin et al 2015 (one of their cites). The idea is pretty obvious: of course you'd like to learn the hyperparameters like they were any other parameter. But that's easier said than done. The problem is that, well, to backpropagate through a hyperparameter, you would need to, say, track how it affects every iteration throughout the entire training run, rather than simply tracking one parameter through a single iteration on a single datapoint. And it's difficult enough to do gradient descent on a single hyperparameter, so it hardly helps to start talking about doing entire stacks! If you can't really do one, doing ad infinitum probably isn't going to work well either. If you look at their experiment, they're doing a 1 hidden-layer FC NN on MNIST. (Honestly, I'm a little surprised that stacking hyperparams works even on that small a problem.)
  • Mehdi2277 7y ago
    Reading the paper, instead of computing the gradient across the training run they compute the gradient for a hyper parameter after each batch and update the hyper parameter then. That keeps the computational cost pretty light. Stacking gradient descent 50 times on itself only costs about double normal gradient descent. That's specific to their experiment, but when done on bigger models the added cost for computing the hyperparameter derivatives should become a smaller fraction. I'm surprised by the lack of any bigger experiments (imagenet would be nice, but even cifar10 would help) given how computationally light this was. Also surprised that an 11ish author stanford paper did their experiments on 1 cpu.
• CardenB 7y ago
  See also: https://news.ycombinator.com/item?id=21142456 https://news.ycombinator.com/item?id=21142456 Has been done before
• mlthoughts2018 7y ago
  This is just Bayesian modeling. Move the parameters into the model, push the hyperparameters into a prior. Move the hyperparameters into the model, push the hyperhyperparameters into a hyperprior, etc. Choosing hyperparameters from minor tuning or literature review or domain expertise or whatever is just a form of creating a prior.
• tagh 7y ago
  This is already done in Gaussian process classification / regression