3 ms·
I would say, to be succinct, that the main trick in ML is Occam's Razor (http://en.wikipedia.org/wiki/Occam%27s_razor http://en.wikipedia.org/wiki/Occam%27s_raz
by khawkins 13y ago
I would say, to be succinct, that the main trick in ML is Occam's Razor (http://en.wikipedia.org/wiki/Occam%27s_razor http://en.wikipedia.org/wiki/Occam%27s_razor).
It has been found that, for most problems, a simple model which well represents previous experience should be accepted instead of a more complex one with marginally better representation. I would claim that the reason this generally works is an empirical discovery, as opposed to a mathematical result, but probably has philosophical implications in its success.
- JASchilz 13y agoCheck out Bayesian Model Selection. It's the mathematical expression of Occam's Razor.
- khawkins 13y agoMy point is that it shows up everywhere, just in different forms. Sparse coding has a penalty for large basises. Gaussian process regression tunes the density of its representation using Bayesian model Selection. SVMs have a slack parameter which dictates how many errors you'll tolerate to reduce the number of hyperplanes.
- JASchilz 13y agoI apologize, my reply was aimed too low.
- PeterisP 13y agoThis is often represented explicitly within the model math by adding a penalty factor that marks more complex models as worse and thus also optimizes simplicity in addition to accuracy.