5 ms·
It is relatively clear why it works beyond the training and validation set: What is being approximated is a smooth function, which in the case of a classificati
by orbifold 8y ago
It is relatively clear why it works beyond the training and validation set: What is being approximated is a smooth function, which in the case of a classification task is a function from the space of things to be classified (images of a certain size) to the n-simplex, where n is the number of classes. Then the preimage theorem tells you that over a regular point of this smooth map lies a codimension n submanifold in the space of things to be classified. That in turn can be interpreted as the submanifold of all things that look like the class you are assigning, especially close to the corners of the n-simplex (being a regular point is an open condition). In short: Because the map is constructed to be smooth it will make sense beyond whatever the training / validation data was. Note that this does not guarantee that it has learned something reasonable about the dataset, just that it will have found some way to smoothly separate it into different components.
- ohfunkyeah 8y agoSome days I think I'm smart, then I read stuff like this. I wish there was an EILI5 Plugin for chrome that I could use to tag any paragraph and someone would come along and break it down for me. I'd do it for your comment!
- placebo 8y agoYou might be confusing "smart" with "knowledgeable". There's no way to be smart about concepts you haven't yet learned. And the ELI5 idea is cool - just need to find a business model :)
- yorwba 8y agoMaybe the ELI5 plugin could be reasonably approximated by adding links to all phrases with corresponding Wikipedia articles, like: https://en.wikipedia.org/wiki/Validation_set https://en.wikipedia.org/wiki/Validation_set https://en.wikipedia.org/wiki/Smooth_function https://en.wikipedia.org/wiki/Smooth_function https://en.wikipedia.org/wiki/N-simplex https://en.wikipedia.org/wiki/N-simplex https://en.wikipedia.org/wiki/Preimage_theorem https://en.wikipedia.org/wiki/Preimage_theorem https://en.wikipedia.org/wiki/Codimension https://en.wikipedia.org/wiki/Codimension https://en.wikipedia.org/wiki/Submanifold https://en.wikipedia.org/wiki/Submanifold I guess it depends on whether you're willing to read a textbook's worth of Wikipedia articles. (There is no shortcut. Actual 5-year-olds also require a few years until they can understand advanced mathematics.)
- srean 8y agoThe question is why dont kernel methods (with appropriate smoothness inducing kernels) and relatedly Gaussian processes do as well. I am being deliberately presumptuous, it is not a proven result that RKHS based methods would be dominated by DNNs. These two approches, RKHS and DNNs arent that fundamentally different, and we have known that even before the recent explosion of interest in DNNs. That infinite DNNs converge to Gaussian processes is an old result. Of course there are differences, once you choose a kernel the feature map is set in stone, on the otherhand DNNs can search over the space of feature maps. It is likely that the function that a DNN converged to was already in the Hilbert space associated with the kernel, but that does not mean one would converge to it in a finite amount of training data. Once one goes to infinite dimensions all separable Hilbert spaces are the same (up to isomorphisms) but with finite data it matters what basis / kernel one chooses.
- amelius 8y agoYes, but these things should be in the guide, or at least with lower-dimensional framing to make it understandable.