3 ms·
The point that combining kernels lets you compose arbitrarily complex functions could come in the beginning. That's sort of why GPs are so exciting in the first
by doctorpangloss 8y ago
The point that combining kernels lets you compose arbitrarily complex functions could come in the beginning. That's sort of why GPs are so exciting in the first place.
In particular, there's a kernel we could call a "change point," which is a way to have a totally different model fitted before a point in time versus after. It's used frequently in Automatic Statistician fitted models (https://automaticstatistician.com/examples/ https://automaticstatistician.com/examples/) which in my opinion are the state of the art of what you can use GPs for. They also developed a "LISP" like representation of the GP kernels, which lets them sample functions, fit them, and publish the simplest ones.
You can see an example of, "Create GP functions and try fitting them" here: https://github.com/probcomp/notebook/blob/master/tutorials/extrapolation-with-gp.ipynb https://github.com/probcomp/notebook/blob/master/tutorials/e... . Near the end of the notebook, you can see the "source code" of the fitted GP function. Note this implementation supports change points, but does not happen to need them on the sample data.
But generally, I wonder in which applications GP shines.
On the one hand, with its emphasis on time series data, GP has a lot to offer to finance, especially options pricing. On the other hand, GP boils down to "the near future looks a lot like the near past," which most people already know.
Clearly, what we want to know is: when will change points occur? Whoever cracks that nut has found GP its breakthrough application.
It should be clear from the example that some form of fitting over the generation of "Gaussian Process Programs" is a good first step.
- man-and-laptop 8y agoMaybe one usage could be in Thompson sampling. You're trying to guess the maximum of a function, so you sample a random function from the GP, find its maximum, test it out, get back a y value, and then update your GP. Maybe that's what the above comment meant by Bayesian optimisation.
- Zephyr314 8y agoThis is close to how Bayesian optimization usually works, except you can sample many functions from the GP at once and apply different "acquisition functions" instead of just looking for the maximum. Classic examples are "Expected Improvement" (which point is expected to beat the current best point by the most) or "Probability of Improvement" (which point has the best odds of beating the current best point). You then sample and repeat. I gave a high level talk that walks through this here [0]. A deeper dive into acquisition functions can be found here [1] and here [2]. [0]: https://www.youtube.com/watch?v=J6UcAdH54RE&list=PLbSwfqjMfjGc6Z-WP9VYBCZsHHyOImabQ https://www.youtube.com/watch?v=J6UcAdH54RE&list=PLbSwfqjMfj... [1]: https://sigopt.com/blog/expected-improvement-vs-knowledge-gradient/ https://sigopt.com/blog/expected-improvement-vs-knowledge-gr... [2]: https://sigopt.com/blog/clustering-applied-to-acquisition-functions/ https://sigopt.com/blog/clustering-applied-to-acquisition-fu... (edit: fixed a typo)
- cafebeen 8y agoOne major application is in geospatial statistics for a variety of fields that need to perform regression of irregularly samples across space. Although it is typically called Kriging, it is mathematically equivalent to Gaussian processes from my understanding: https://en.wikipedia.org/wiki/Kriging https://en.wikipedia.org/wiki/Kriging
- CuriouslyC 8y agoThe killer feature of Gaussian processes for me is the fact that they provide a posterior distribution rather than just a maximum likelihood estimate. This lets you incorporate fancy loss functions, and can be useful to optimize data collection in real world cases where collecting data is expensive and time consuming.
- Zephyr314 8y agoAt SigOpt (YC W15, disclaimer: I'm a founder) we use GPs among other Bayesian and global optimization techniques for black box parameter optimization. In our blog we go over GP intuition [0], likelihoods [1], different kernels [2] (we also have some papers on this [3]), and acquisition functions for applying GPs to optimization [4]. In this talk [5] I go over the fundamentals on how you can apply GPs to this problem as well. [0]: https://sigopt.com/blog/intuition-behind-gaussian-processes/ https://sigopt.com/blog/intuition-behind-gaussian-processes/ [1]: https://sigopt.com/blog/likelihood-for-gaussian-processes/ https://sigopt.com/blog/likelihood-for-gaussian-processes/ [2]: https://sigopt.com/blog/covariance-kernels-for-avoiding-boundaries/ https://sigopt.com/blog/covariance-kernels-for-avoiding-boun... [3]: https://sigopt.com/research/papers/ https://sigopt.com/research/papers/ [4]: https://sigopt.com/blog/expected-improvement-vs-knowledge-gradient/ https://sigopt.com/blog/expected-improvement-vs-knowledge-gr... [5]: https://www.youtube.com/watch?v=J6UcAdH54RE&list=PLbSwfqjMfjGc6Z-WP9VYBCZsHHyOImabQ https://www.youtube.com/watch?v=J6UcAdH54RE&list=PLbSwfqjMfj...