7 ms·

The dirty secret in ML is that logistic regression, SVM, and random forests often work better than deep learning on real problems.

• akhilcacharya 10y ago
  For image classification, CNN's are still the way to go. But creating your own architecture and training your own novel model aren't necessary for most problems anymore it seems because of transfer learning.
  • markovbling 10y ago
    I threw all of the above plus CNNs at MNIST problem and boosted decision trees outperformed CNNs. Granted if I tuned both perfectly, CNNs probably would have outperformed but with defaults and a small amount of parameter search, boosting worked best.
    • akhilcacharya 10y ago
      That's pretty interesting, why are boosted decision trees so effective? I've heard the same meme applied to kaggle competitions (everything is just a way to shove data into xgboost, etc)
      • romaniv 10y ago
        The grandparent post contains a hint of something I've already heard in lectures and from ML practicioners: boosting and random forests are more resilient to improper tuning (to put it another way, they are more universal and work well "out of the box"). Which, BTW, makes them more appealing to me personally. In many real-life cases extra few percent of accuracy matters very little, but ability to just apply something to a problem without much fuss matters a lot.
  • markovbling 10y ago
    I threw all of the above plus CNNs at MNIST problem and boosted decision trees outperformed CNNs. Granted if I tuned both perfectly, CNNs probably would have outperformed but with defaults and a small amount of parameter search, boosting worked best.
• thinkr42 10y ago
  You are absolutely right about that!
• deepnotderp 10y ago
  Uhhh most deep nets for classification or even regression have a regression at the end. You can even use a svm at the end. The power of deep learning is in the feature learning.
  • argonaut 10y ago
    Yes, but using an SVM or whatever at the end of a deep network only really works for the same domains that deep networks work well in already - typically audio/visual data and some language datasets.
    • deepnotderp 10y ago
      Most "simple" datasets like insurance prediction from structured data has been solved already via traditional techniques. The difficulty in machine learning was always in the noisy and unstructured datasets like images.
      • et2o 10y ago
        "solved" I'm going to disagree with you here.
      • argonaut 10y ago
        I'm curious what background you have in ML. "The difficulty in machine learning was always in the noisy and unstructured datasets like images" is truly a baffling statement to make.
• eanzenberg 10y ago
  This is literally how you can distinguish actual machine learners from blog readers. - Deep learn everything! vs. - I took at look at the variable distributions, went with a forest model after transforming some of the data.
  • digitalzombie 10y ago
    > I took at look at the variable distributions, went with a forest model after transforming some of the data. That's what a statistician does too... I wish statistician word is more in vogue than data science or machine learners. Statistic is the discipline of data.
• axplusb 10y ago
  Gradient boosting also often does wonders.