> if you have a "good" "working" "theory of how neural networks operate", what is it a "theory of"? I think we should make the distinction between theory perta

> if you have a "good" "working" "theory of how neural networks operate", what is it a "theory of"? I think we should make the distinction between theory pertaining to a task and theory pertaining to methods that perform (or approximate) the task. Certainly, the former can be incorporated into the latter, so the boundary is fuzzy. Actually, the previous is quite important because known principles of the task can be expressed mathematically and incorporated functionally into the approximation method. In this sense, a network architecture could arise naturally. In fact, it sort of does with anything with a cascade-type pattern. On the other hand, if we're going to talk about the method (networks, in particular) independently of the task, this is more difficult. The question is now: is the network model remarkable in some sense? Meaning: is there some class of functions which are "best" or "more efficiently represented" by network approximations, and what are the properties of the class that make this the case? Yoshua Bengio has touched on this with regards to depth from the point of view of circuit theory, but the argument is basically: "here's a couple circuits which are more efficiently represented by increased depth, therefore deep = good always". It would be more interesting if there was a more rigorous analysis from a function approximation view. Perhaps literature exists on this. I'm not sure - I'm sort of rambling now. > kernel trick plus projection onto feature space, is basically ad-hoc too The choice of kernel - yes I agree, but the driving theory of the method is to maximize the margin, not choose the best kernel. > Creating more complex systems to tackle that tends to fail as people wind-up understanding less and less of their own complex systems. Interesting. Maybe there's something going on with the relationship between entropy and complexity.