3 ms·
Random question from someone who's never touched ML professionally: Can we optimize hyper-params the same way we optimize weights and biases (e.g gradient desc
by brap 4y ago
Random question from someone who's never touched ML professionally:
Can we optimize hyper-params the same way we optimize weights and biases (e.g gradient descent)? Or would that be too expensive (since you have to optimize weights and biases for each hyper-param configuration)?
Of course that might lead to hyper-hyper-params...
- blackbear_ 4y agoIn the general case no because the final metric we care about (eg, accuracy or AUC) is not differentiable (ie, we cannot compute its gradient) with respect to the hyperparameters, especially the discrete ones. However a recent work at this year's NeurIPS [1] did indeed use an outer gradient descent to tune an inner gradient descent so in some cases yours is indeed a good idea that works. [1] https://arxiv.org/abs/1909.13371 https://arxiv.org/abs/1909.13371