5 ms·

I'd just like to note that instead of creating additional animosity between SVMs and deep nets, you could use both together. SVMs with hinge loss can be Yet-ano

I'd just like to note that instead of creating additional animosity between SVMs and deep nets, you could use both together. SVMs with hinge loss can be Yet-another-layer (tm) in your deep net, to be used when it provides better performance.

strebler 9y ago
That's a great point. Fundamentally, if you look at something like a CNN, what it's really doing is producing a feature descriptor based on the input image. One can easily use that feature descriptor in a classic SVM, alongside (or instead of) SoftMax.
- deepnotderp 9y ago
  Yup, in fact, the universal feature extraction is what allows imagenet pretraining to work well on lung cancer images. One nitpick though, ConvNets can absolutely be used to do "thinking" and more than just feature extraction. For example, fully convolutional networks can be extremely competitive with FC-layer based nets.
- JustFinishedBSG 9y ago
  You may be interested by https://arxiv.org/abs/1605.06265 https://arxiv.org/abs/1605.06265 http://papers.nips.cc/paper/5348-convolutional-kernel-networks.pdf http://papers.nips.cc/paper/5348-convolutional-kernel-networ...
sivvy 9y ago
Could you explain in a bit more detail how you would integrate an SVM layer into a DNN? The kernel matrix depends on all samples, while at training time you would only have access to those in the minibatch.
- IanCal 9y ago
  The simplest is to pop it on the top. Run you DNN to reduce your input down to a nicer cleaner smaller dimensional output, then plop an SVM on top for classification.
  - sivvy 9y ago
    Seems like in that case you would train both models separately on different cost functions. By phrasing it as a layer I was expecting both the SVM and the DNN could be trained simultaneously.
    - IanCal 9y ago
      Unless things have changed, one of the key benefits of DNNs was that you trained them layer by layer. You also want to be able to train the DNN on your unlabelled data and the SVM on your much smaller labelled set.
JustFinishedBSG 9y ago
Yes https://arxiv.org/abs/1605.06265 https://arxiv.org/abs/1605.06265 http://papers.nips.cc/paper/5348-convolutional-kernel-networks.pdf http://papers.nips.cc/paper/5348-convolutional-kernel-networ...

strebler 9y ago

That's a great point. Fundamentally, if you look at something like a CNN, what it's really doing is producing a feature descriptor based on the input image. One can easily use that feature descriptor in a classic SVM, alongside (or instead of) SoftMax.

deepnotderp 9y ago
Yup, in fact, the universal feature extraction is what allows imagenet pretraining to work well on lung cancer images. One nitpick though, ConvNets can absolutely be used to do "thinking" and more than just feature extraction. For example, fully convolutional networks can be extremely competitive with FC-layer based nets.
JustFinishedBSG 9y ago
You may be interested by https://arxiv.org/abs/1605.06265 https://arxiv.org/abs/1605.06265 http://papers.nips.cc/paper/5348-convolutional-kernel-networks.pdf http://papers.nips.cc/paper/5348-convolutional-kernel-networ...

sivvy 9y ago

Could you explain in a bit more detail how you would integrate an SVM layer into a DNN? The kernel matrix depends on all samples, while at training time you would only have access to those in the minibatch.

IanCal 9y ago
The simplest is to pop it on the top. Run you DNN to reduce your input down to a nicer cleaner smaller dimensional output, then plop an SVM on top for classification.
- sivvy 9y ago
  Seems like in that case you would train both models separately on different cost functions. By phrasing it as a layer I was expecting both the SVM and the DNN could be trained simultaneously.
  - IanCal 9y ago
    Unless things have changed, one of the key benefits of DNNs was that you trained them layer by layer. You also want to be able to train the DNN on your unlabelled data and the SVM on your much smaller labelled set.

JustFinishedBSG 9y ago

Yes https://arxiv.org/abs/1605.06265 https://arxiv.org/abs/1605.06265 http://papers.nips.cc/paper/5348-convolutional-kernel-networks.pdf http://papers.nips.cc/paper/5348-convolutional-kernel-networ...