3 ms·
This isn't shooting completely in the dark, and obviously there's clear motivation, but motivation, inspiration and engineering are still not theory. Let me pu
by pron 10y ago
This isn't shooting completely in the dark, and obviously there's clear motivation, but motivation, inspiration and engineering are still not theory.
Let me put it another way: that paper reads like a technical report or a paper in an experimental science. It does not all resemble a "normal" academic CS or math paper. That's fine, maybe that's the stage this discipline is at, but I was surprised to see that there's still no new theory since when I worked with NNs in the 90s.
- zodiac 10y agoI agree with this. I think it will be a while before we get a decent theory, though, and it's not because no one is trying, but it seems too hard to me. For instance, the way I see it, we don't even know very well how the low-dimensional space of (for example) "naturally occurring pictures" is embedded into the high-dimensional space of "all possible pictures". So we can't even precisely state what we want our NNs to do, much less why they work well. Put another way, if we ask a mathematician "so why is it that when I use gradient descent on this architecture, I perform reasonably well on unseen examples?", she would reply, "ok, what distribution did you draw your training and unseen examples from?", and we can't answer that.