3 ms·
> and it's not even mentioned that you can optimize an ML model without using the gradient. Can you, though? How does the outcomes compare to gradient based me
by xyzzyz 6y ago
> and it's not even mentioned that you can optimize an ML model without using the gradient.
Can you, though? How does the outcomes compare to gradient based methods?
- cgearhart 6y agoIn general GFO is not a good alternative when there are lots of model parameters or when it’s cheap to get the gradient. GFO could be used to train models, but kinda in the same way that you could find your way to Cleveland by just wandering around long enough. It sure would help a lot if you had a compass.
- xyzzyz 6y agoYeah, that's what I thought. For deep learning models, when your parameter space has many thousands dimensions, it's really cheap to use gradient-based methods, when you can use stochastic gradient descent and backpropagation to quickly calculate good estimates of gradient. I simply don't see why eschewing this structure can lead to anything competitive.