3 ms·
We train deep learning models with SGD, which is in fact a form of search -- in a high-dimensional parameter space. The purpose of SGD is to find parameter val
by cs702 3y ago
We train deep learning models with SGD, which is in fact a form of search -- in a high-dimensional parameter space.
The purpose of SGD is to find parameter values that minimize a training loss.
- spmurrayzzz 3y agoI was thinking along similar lines, it does smell like search to me at its core. But do you think there is a meaningful differentiation for something more nuanced like AdamW? Does the latter fall into the "brute force search" category too though it tends to converge faster (and arguably more optimally than SGD per Jeremy Howard's observations)?