4 ms·
I used simulated annealing in my previous work doing molecular dynamics- it was great for not getting your protein trapped in a local minima, which gradient des
by dekhn 3y ago
I used simulated annealing in my previous work doing molecular dynamics- it was great for not getting your protein trapped in a local minima, which gradient descent is prone to doing. I got excellent results for very little work.
I asked my intern, who was knowledgeable in deep networks as well as molecular stuff, "it looks like ML training mainly does gradient descent, how can that work, don't you get stuck in local minima?" and they said "loss functions in ML are generally believed to be bowl-shaped" and I've been wondering how that could be true.
It's interesting to read up on the real-world use of annealing for steel - it's quite intersting how you can change steel properties through heat treatment. Want it really strong? Quench it fast, that will lock it into an unstable structuer that's still strong. Quench it slow, it will find a more stable minimum, and be more ductile.
- pclmulqdq 3y agoTheir explanation is relatively incomplete, I think. The best explanation I can come up with regarding why gradent descent works is that the optimization space is extremely high-dimensional, which has the tendency to force functions that would look weird in low dimensions to be more well-behaved. Points taken randomly in high-dimensional space also tend to be closer together. The numerical benefits of dimensionality are my belief as to why the "stack more layers" crowd has been so successful.
- imtringued 3y agoHigh dimensionality allows gradient descent to make progress independently in each dimension. When you get stuck in one dimension, you make progress in another dimension until you can progress in the previously blocked dimension. Also, even if you do get stuck in X dimensions, you are unlikely to significantly reduce that X by finding a better starting point and if there is redundancy, then it doesn't matter that you have the local minima in one dimension but not the other dimension. The end result is that the quality differences between the local minima end up as insignificant. They are sufficiently good compared to the global minima.
- deleted 3y ago[deleted]
- pama 3y agoModern DL architectures are complicated. If you want to gain insight if a loss function is convex (bowl shaped), you can calculate the second derivatives matrix (Hessian) at arbitrary points and test that all its eigenvalues are non negative. If you have a simple linear regression then that’s true, but for modern DL architectures things are not always convex and of course Hessians in very high dimensions are unrealistic. A simpler numerical test is to take lines connecting two arbitrary points of the loss function and showing that the value of the function along the line of the function variable connecting the two points is always lower than or equal to the value of the line connecting the two points. It is also hard to make this test rigorous in a practical high dimensional setting but one could be lucky and eg connect local minima to show that the function is not convex. The practical reason people didn’t use annealing on the DL problems at scale has been the satisfactory empirical results of the simpler/faster methods.