3 ms·
I hope to see gradient-free/black box optimization algorithms gain more popularity in machine learning. I'm always surprised when I see a review of ML optimizat
by optimalsolver 6y ago
I hope to see gradient-free/black box optimization algorithms gain more popularity in machine learning. I'm always surprised when I see a review of ML optimization methods, and it's not even mentioned that you can optimize an ML model without using the gradient.
Also, love or hate Python, there's almost always a library for whatever it is you want to do.
- xyzzyz 6y ago> and it's not even mentioned that you can optimize an ML model without using the gradient. Can you, though? How does the outcomes compare to gradient based methods?
- cgearhart 6y agoIn general GFO is not a good alternative when there are lots of model parameters or when it’s cheap to get the gradient. GFO could be used to train models, but kinda in the same way that you could find your way to Cleveland by just wandering around long enough. It sure would help a lot if you had a compass.
- xyzzyz 6y agoYeah, that's what I thought. For deep learning models, when your parameter space has many thousands dimensions, it's really cheap to use gradient-based methods, when you can use stochastic gradient descent and backpropagation to quickly calculate good estimates of gradient. I simply don't see why eschewing this structure can lead to anything competitive.