3 ms·
Thanks for reading, I'm glad you enjoyed it. You are correct that there are few texts that discuss real-world examples of reproducing kernel Hilbert spaces wit
by mccourt 11y ago
Thanks for reading, I'm glad you enjoyed it.
You are correct that there are few texts that discuss real-world examples of reproducing kernel Hilbert spaces without a significant amount of overhead to get there. In reality, I'm not sure there is an easy way to talk about reproducing kernel Hilbert spaces without all that functional analysis, which is a tough topic. I would argue that's probably the greatest thing holding back kernels - their high barrier to entry.
I'll spend one line here to mention my book (kernel-based approximation methods in Matlab) and also say that it is pretty heavy on the theory; the second half of the book deals with reproducing kernels as used in a variety of fields, but getting there requires a lot of very tough math.
Lemme take a moment to list places (some more real-world than others) where I know that reproducing kernels pop up - and anyone else reading this please feel free to add to this list:
1) Approximation theory - My bread and butter and the most important role for kernels as far as I'm concerned. When you have scattered data (especially in higher dimensions) and you want to interpolate and predict unobserved values, reproducing kernels give you several very nice guarantees. Through this, they also pop up in computer vision or surface reconstruction from point cloud data.
2) Spatial statistics - They actually play an identical role in kriging as they do in approximation theory, albeit for totally different reasons. People who are doing remote sensing or topological mapping may use reproducing kernels (or software that uses these kernels.
3) Machine learning - Both RBF networks and support vector machines use reproducing kernels, although in two different ways. The mechanism by which they are used in support vector machines (feature maps) is difficult for me to understand which is why I don't talk about them much. They are still a very important field that uses kernels.
4) Differential equations - The same theory that makes reproducing kernels work for approximation theory also works for numerically solving differential equations. You could lump stuff regarding Green's functions in here too, although those may not be reproducing kernels.
5) Numerical integration - Especially for integration in higher dimensions, basically all the theory of convergence for quasi Monte Carlo methods relies on reproducing kernel Hilbert spaces.
Maybe the best thing I can point you to is the same place where I first learned about this stuff: math.iit.edu/~fass/590. That's an old (recently updated) class website from a class I took at the Illinois Institute of Technology when I was an undergrad there. Dr. Fasshauer does an outstanding job putting notes and slides for his class together.
If you're feeling adventurous, I would recommend our book, but a good starting point would be his class notes, which I still turn back to often and are written to introduce graduate students (who know nothing about kernels) to the topic. I'll work on putting together a list of good application-oriented sources as well.
Thanks for your interest! Did you have a specific example/topic of kernels from data analysis that piqued your interest? I could look into more targeted content.
- brianchu 11y agoHonestly, the biggest thing holding back kernels is deep learning (unfortunately?).
- mccourt 11y agoThat may be. And I don't think kernels will ever be a hot topic. But part of that is that they are a very old topic (you can see Gauss referring to them in slide 22 of http://math.iit.edu/~fass/590/notes/Notes590_Ch1.pdf http://math.iit.edu/~fass/590/notes/Notes590_Ch1.pdf). I think that kinda prevents funding agencies from pushing too hard for research into them, since there is a more likely quick return on investment into a new topic. And I don't begrudge them that - funding should be given to people with new ideas which have the potential to revolutionize things. I'd also mention that there are people trying to bridge the gap between how GPs work and the benefits of neural nets (http://arxiv.org/abs/1502.05700 http://arxiv.org/abs/1502.05700). Probably another thing holding them back is their high barrier to entry - you can't do much with the theory of kernels unless you know statistics and functional analysis (and of course numerical analysis and linear algebra,) which is one of the reasons I had difficulty finding research students. To draw a parallel to numerical analysis (where I am much more comfortable than machine learning,) this is why finite element methods are much more popular than, say, boundary element methods for PDEs, despite the fact that boundary element methods are better in many circumstances. But when implementing things, especially in industry, simplicity and robustness count for something and those are not strong points of kernels. But kernels haven't gone anywhere yet, and they've been fundamental to analysis since at least David Hilbert, so they are not going anywhere any time soon. Deep learning ... we'll see. Maybe it has the legs to stick around or maybe it'll be swept aside by the next hot thing. 8 years ago all the research funding was going into uncertainty quantification, 5 years ago compressed sensing was the hot idea (thanks Terry Tao,) and now deep learning is going to light the world on fire. All of those are, and should be, hotter topics than kernels, but we'll see what we're talking about in 5 years. It won't be kernels (unless something crazy happens and I become the leader of the US, in which case I have a great idea for a new reality show) but I don't know that it will be deep learning either.