4 ms·
I think the reasoning still holds. If the dimensions are reasonably independent of each other, the probability of a local minimum occurring in any kind of high-
by phreeza 10y ago
I think the reasoning still holds. If the dimensions are reasonably independent of each other, the probability of a local minimum occurring in any kind of high-dimensional surface falls rapidly as the number of dimensions increases. And I would argue that machine learning methods are a pretty high-dimensional search space, describing them in a single 'method of attack' dimension is probably not right.
- jal278 10y agoBut by that line of reasoning, shouldn't we never hit dead ends in AI research at all -- why has AI progress been so difficult, then? Wouldn't any field of research with many dimensions of variation never get stuck on its path towards its ultimate goals, ever? Couldn't different objective functions be structurally more difficult than others to optimize? No matter how high-dimensional the search-space, trying to create a gaming laptop in the middle ages would have been a pretty frustrating experience.
- phreeza 10y agoBecause there are plenty of saddle points. Gradient descent slows down quite a bit at those, too.