4 ms·
Author here, thanks for the comment! While this is an interesting read (and NTMs are great), it is not particularly relevant to the model I describe or this pa
by kennethfriedman 10y ago
Author here, thanks for the comment!
While this is an interesting read (and NTMs are great), it is not particularly relevant to the model I describe or this paper.
I think you are misunderstanding my generality of my paper: I am not discussing a particular method of deep learning. I am using the idea of gradient descent as an metaphor for the field of AI, itself.
As described in the second paragraph of the "Gradient Descent" section, this analogy is not high dimensional. In fact, it is only three dimensional: distance the field is from General AI, time, and a hypothetical "method of attack".
- phreeza 10y agoI think the reasoning still holds. If the dimensions are reasonably independent of each other, the probability of a local minimum occurring in any kind of high-dimensional surface falls rapidly as the number of dimensions increases. And I would argue that machine learning methods are a pretty high-dimensional search space, describing them in a single 'method of attack' dimension is probably not right.
- jal278 10y agoBut by that line of reasoning, shouldn't we never hit dead ends in AI research at all -- why has AI progress been so difficult, then? Wouldn't any field of research with many dimensions of variation never get stuck on its path towards its ultimate goals, ever? Couldn't different objective functions be structurally more difficult than others to optimize? No matter how high-dimensional the search-space, trying to create a gaming laptop in the middle ages would have been a pretty frustrating experience.
- phreeza 10y agoBecause there are plenty of saddle points. Gradient descent slows down quite a bit at those, too.