3 ms·

My ML PhD friends say that SVM are popular in academic circles in part be cause you can prove things on them. They are in the NN camp when it comes to solving

My ML PhD friends say that SVM are popular in academic circles in part be cause you can prove things on them. They are in the NN camp when it comes to solving problems. I like the basic idea. It feels like a nonlinear relative of PCA (Principal Component Analysis). I also haven't seen NN used for outlier detection.

tel 13y ago
SVM comes from a theory called Vapnik–Chervonenkis dimension which talks about ways to balance best learner performance against likelihood of actually achieving that theoretical optimum. It's very nice theory. The nonlinear stuff is called the "kernel trick" and is essentially a recognition that an SVM's operation is depends on the data exclusively through measurements of how far apart the n^2 data points are from one another. Once you realize that you can twist the definitions around so as to insert whatever kind of non-linear notion of "distance" you like. Many ML algorithms can be augmented by the kernel trick. Finally, SVM's geometry allows you to think of it as a relatively nice optimization problem. Compare this to many other ML algorithms which have only heuristic optimization methods. It makes the algorithm surprisingly fast.
lightcatcher 13y ago
I'm not sure if its been done before, but neural networks could be used for outlier detection. Autoencoders can also be thought of as a form on nonlinear PCA, and the LISA group in Montreal has actually proved that the reconstruction error from a contractive or denoising autoencoder for a data point serves as an estimator of the probability density under that point.
dlwh 13y ago
Actually, NNs (specifically auto-encoders) are more like a nonlinear PCA. PCA can be seen as an "information bottleneck" where you try to represent an N dimensional space as a K < N dimensional space such that your ability to reconstruct the original space is maximized (that is, the "reconstruction error" is minimized.) Autoencoders are precisely the same idea, except that you use nonlinear transformations (e.g. softmax or tanh) in addition to linear transformations (matrix multiplications). You can also use multiple layers in an autoencoder, just like any neural net. (The linearness of PCA makes multiple layers useless. You can always replace multiple layers with a single layer.)

tel 13y ago

SVM comes from a theory called Vapnik–Chervonenkis dimension which talks about ways to balance best learner performance against likelihood of actually achieving that theoretical optimum. It's very nice theory. The nonlinear stuff is called the "kernel trick" and is essentially a recognition that an SVM's operation is depends on the data exclusively through measurements of how far apart the n^2 data points are from one another. Once you realize that you can twist the definitions around so as to insert whatever kind of non-linear notion of "distance" you like. Many ML algorithms can be augmented by the kernel trick. Finally, SVM's geometry allows you to think of it as a relatively nice optimization problem. Compare this to many other ML algorithms which have only heuristic optimization methods. It makes the algorithm surprisingly fast.

lightcatcher 13y ago

I'm not sure if its been done before, but neural networks could be used for outlier detection. Autoencoders can also be thought of as a form on nonlinear PCA, and the LISA group in Montreal has actually proved that the reconstruction error from a contractive or denoising autoencoder for a data point serves as an estimator of the probability density under that point.

dlwh 13y ago

Actually, NNs (specifically auto-encoders) are more like a nonlinear PCA. PCA can be seen as an "information bottleneck" where you try to represent an N dimensional space as a K < N dimensional space such that your ability to reconstruct the original space is maximized (that is, the "reconstruction error" is minimized.) Autoencoders are precisely the same idea, except that you use nonlinear transformations (e.g. softmax or tanh) in addition to linear transformations (matrix multiplications). You can also use multiple layers in an autoencoder, just like any neural net. (The linearness of PCA makes multiple layers useless. You can always replace multiple layers with a single layer.)