I agree with this. I think it will be a while before we get a decent theory, though, and it's not because no one is trying, but it seems too hard to me. For in

I agree with this. I think it will be a while before we get a decent theory, though, and it's not because no one is trying, but it seems too hard to me. For instance, the way I see it, we don't even know very well how the low-dimensional space of (for example) "naturally occurring pictures" is embedded into the high-dimensional space of "all possible pictures". So we can't even precisely state what we want our NNs to do, much less why they work well. Put another way, if we ask a mathematician "so why is it that when I use gradient descent on this architecture, I perform reasonably well on unseen examples?", she would reply, "ok, what distribution did you draw your training and unseen examples from?", and we can't answer that.