3 ms·

Also, one huge reason why we might "limit ourselves to small datasets" is because that's the only way we can compare many algorithms. Outside training data has

by gcr 13y ago

Also, one huge reason why we might "limit ourselves to small datasets" is because that's the only way we can compare many algorithms. Outside training data has a huge influence on algorithm accuracy. For example, since Facebook has access to bajillions of face images that they could use to train their classifier, and since they can't share those images with the rest of the community, it's unclear whether their excellent performance is because they have gobs of data or whether it's really a better algorithm. I bet that a simple algorithm like a naive SVM might do leagues better if we could train it on Facebook's (hidden!) dataset and test on LFW. It just isn't reproducible---a measuring stick is only meaningful if everyone uses the same measuring stick.

tormeh 13y ago
Isn't small datasets the reason for using SVMs? You are, after all, operating in linear space preprocessed with a kernel trick. Far less expressive than a neural network.
- gcr 13y ago
  Well, isn't it a bit awkward when dataset size is what forces you to select a certain algorithm that you otherwise wouldn't use? "Gee, I would love to train a neural network, but it's just so much data and I'm on a deadline; maybe I should just use an SVM and hope for the best ... ..." That's one of the big surprises behind deep learning these days: it's now feasible to do things like "train a big neural network on bunches of images" in a sensible time. It's an optimization thing as much as it is a machine learning thing, in my opinion.
  - tormeh 13y ago
    I didn't mean that training time is the constraint. It's that when training a more general algorithms (ANN) your hypothesis space has more dimensions than when training a more specialized one (SVM). You therefore need more data to train an ANN than a SVM. The reason for choosing SVM is not that your dataset is too big for ANN, it's that it's too small for ANN.

by gcr 13y ago

tormeh 13y ago

Isn't small datasets the reason for using SVMs? You are, after all, operating in linear space preprocessed with a kernel trick. Far less expressive than a neural network.

gcr 13y ago
Well, isn't it a bit awkward when dataset size is what forces you to select a certain algorithm that you otherwise wouldn't use? "Gee, I would love to train a neural network, but it's just so much data and I'm on a deadline; maybe I should just use an SVM and hope for the best ... ..." That's one of the big surprises behind deep learning these days: it's now feasible to do things like "train a big neural network on bunches of images" in a sensible time. It's an optimization thing as much as it is a machine learning thing, in my opinion.
- tormeh 13y ago
  I didn't mean that training time is the constraint. It's that when training a more general algorithms (ANN) your hypothesis space has more dimensions than when training a more specialized one (SVM). You therefore need more data to train an ANN than a SVM. The reason for choosing SVM is not that your dataset is too big for ANN, it's that it's too small for ANN.