3 ms·
Their explanation is relatively incomplete, I think. The best explanation I can come up with regarding why gradent descent works is that the optimization space
by pclmulqdq 3y ago
Their explanation is relatively incomplete, I think. The best explanation I can come up with regarding why gradent descent works is that the optimization space is extremely high-dimensional, which has the tendency to force functions that would look weird in low dimensions to be more well-behaved. Points taken randomly in high-dimensional space also tend to be closer together.
The numerical benefits of dimensionality are my belief as to why the "stack more layers" crowd has been so successful.
- imtringued 3y agoHigh dimensionality allows gradient descent to make progress independently in each dimension. When you get stuck in one dimension, you make progress in another dimension until you can progress in the previously blocked dimension. Also, even if you do get stuck in X dimensions, you are unlikely to significantly reduce that X by finding a better starting point and if there is redundancy, then it doesn't matter that you have the local minima in one dimension but not the other dimension. The end result is that the quality differences between the local minima end up as insignificant. They are sufficiently good compared to the global minima.
- deleted 3y ago[deleted]