3 ms·
Would this work on DL models?
by amelius 3y ago
Would this work on DL models?
- haltist 3y agoGradient free methods work on any optimization problem, including deep networks, because gradient free methods do not require a smooth parametrization of the search space. For deep networks the parameters are on a smooth manifold and that's why gradient based optimizers are used to find near optimal solutions/parameters, i.e. the search space has the structure of a smooth manifold so the parameters can be updated with gradient descent.
- wenc 3y agoIt would, but it would be super inefficient and slow. For smooth functions, gradient information gives more information about the search terrain. The more exact your gradient information, the higher quality your iterations become. On the other gradients aren't defined everywhere (e.g. non-smooth functions like abs(x), max(0,x), or discontinuous/discrete integer variables like x=1,2,3,4...), and gradient based methods can land on saddle points. This is where derivative free optimization (DFO) can help. The downside is they converge far, far, far slower than gradient based methods. It's unlike DFO would scale to what is needed to solve even basic DL models.