5 ms·

I've always found it curious that Neural Networks get so much hype when xgboost (gradient boosted decision trees) is by far the most popular and accurate algori

I've always found it curious that Neural Networks get so much hype when xgboost (gradient boosted decision trees) is by far the most popular and accurate algorithm for most Kaggle competitions. While neural networks are better for image processing types of problems, there are a wide variety of machine learning problems where decision tree methods perform better and are much easier to implement.

deleted 10y ago
[deleted]
nojvek 10y ago
Any good links you recommend learning xgboost? I've never quite figured out how they work.
- deleted 10y ago
  [deleted]
- BickNowstrom 10y ago
  http://xgboost.readthedocs.io/en/latest/model.html http://xgboost.readthedocs.io/en/latest/model.html http://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf http://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf https://www.youtube.com/watch?v=wPqtzj5VZus https://www.youtube.com/watch?v=wPqtzj5VZus Trevor Hastie - Gradient Boosting Machine Learning https://www.youtube.com/watch?v=sRktKszFmSk https://www.youtube.com/watch?v=sRktKszFmSk Ensembles (3): Gradient Boosting, Ihler
photoJ 10y ago
There are lots of reasons, principally among them that Kaggle comps problems represent only one tiny fraction of ML problems. On general purpose ML, with out much time rigging a optimal solution, RF/Xgboost will preform better. But in many problems, ie vision, DL is vastly superior. The other important point is that there many new opportunities for researchers regarding DL, where statistical ML approach on supervised problems is a much more well established field.
- photoJ 10y ago
  Also note that current "leader board" standard on MNIST is 99.8, not 99% he compares against. https://en.wikipedia.org/wiki/MNIST_database https://en.wikipedia.org/wiki/MNIST_database
zebrafish 10y ago
I think the hype around CNN is the NLP aspect of it as it relates to AI. If you can hammer down NLP and translate voice or text to computer-legible commands, you've really improved the user experience. On the other side of the CNN coin is the image recognition that's getting a lot of hype from the self driving auto crowd. I think any data scientist worth their salt understands how the different algorithms stack up against each other. You wouldn't use xgboost for a computer vision problem just like you wouldn't use CNN for a tabular data problem.
BickNowstrom 10y ago
The hype for neural networks is deserved. Some major contributions to the field resulted in increases in accuracy for fields like NLP, computer vision, structured data, machine translation, style transfer, etc. XGBoost did not change much from the "Greedy function approximation: A gradient boosting machine." paper, but uses a few tricks to be much much faster, allowing for better tuning. XGBoost is popular for structured data competitions on Kaggle. Even there: The winner is often an ensemble of XGBoost and Keras. And some structured data competitions are won by neural nets alone: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st-place-interview/ http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it... and https://www.kaggle.com/c/higgs-boson/discussion/10425 https://www.kaggle.com/c/higgs-boson/discussion/10425 (Neural nets won the Higgs Boson Detection Challenge, where XGBoost was introduced) I'd say Tensorflow/Keras can handle a wider variety of problems, with the same, or improved accuracy, than tree-based methods can. NN's do well on structured problems (the domain of tree-based methods), but also own computer vision and, increasingly, NLP. I agree on the pitfalls of applying neural nets vs. forests. It is true that tree-based methods are academically a bit out of vogue: The exciting stuff is happening in the neural network space. You have more chance of getting published with deep learning (this used to be the other way around).
- throw_away_777 10y ago
  I agree that neural nets are state-of-the-art and do quite well on certain types of problems (NLP and vision, which are important problems). But a lot of data is structured (sales, churn, recommendations, etc), and it is so much easier to train an xgboost model than a neural net model. You need a very expensive computer or expensive cloud computing to train neural nets, and even then it is not easy. Ease of implementation is an important factor that gets overlooked in academia. And on non-NLP and non-image datasets, usually the single best Kaggle model is an xgboost model, which was probably developed in 1/10th the time it took to make a good neural net model. Xgboost has come a long way since it was first introduced, with early stopping being an example of a significant improvement.
  - photoJ 10y ago
    How can you say that ease of implementation is overlooked in academia, when academia created the exact tools your are speaking of?
sgt101 10y ago
Because in the real world we need robust classifiers not optimised ones. Problem : robust && optimized are very vague terms in ML.

deleted 10y ago

[deleted]

nojvek 10y ago

Any good links you recommend learning xgboost? I've never quite figured out how they work.

deleted 10y ago
[deleted]
BickNowstrom 10y ago
http://xgboost.readthedocs.io/en/latest/model.html http://xgboost.readthedocs.io/en/latest/model.html http://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf http://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf https://www.youtube.com/watch?v=wPqtzj5VZus https://www.youtube.com/watch?v=wPqtzj5VZus Trevor Hastie - Gradient Boosting Machine Learning https://www.youtube.com/watch?v=sRktKszFmSk https://www.youtube.com/watch?v=sRktKszFmSk Ensembles (3): Gradient Boosting, Ihler

photoJ 10y ago

There are lots of reasons, principally among them that Kaggle comps problems represent only one tiny fraction of ML problems. On general purpose ML, with out much time rigging a optimal solution, RF/Xgboost will preform better. But in many problems, ie vision, DL is vastly superior. The other important point is that there many new opportunities for researchers regarding DL, where statistical ML approach on supervised problems is a much more well established field.

photoJ 10y ago
Also note that current "leader board" standard on MNIST is 99.8, not 99% he compares against. https://en.wikipedia.org/wiki/MNIST_database https://en.wikipedia.org/wiki/MNIST_database

zebrafish 10y ago

I think the hype around CNN is the NLP aspect of it as it relates to AI. If you can hammer down NLP and translate voice or text to computer-legible commands, you've really improved the user experience. On the other side of the CNN coin is the image recognition that's getting a lot of hype from the self driving auto crowd. I think any data scientist worth their salt understands how the different algorithms stack up against each other. You wouldn't use xgboost for a computer vision problem just like you wouldn't use CNN for a tabular data problem.

BickNowstrom 10y ago

The hype for neural networks is deserved. Some major contributions to the field resulted in increases in accuracy for fields like NLP, computer vision, structured data, machine translation, style transfer, etc. XGBoost did not change much from the "Greedy function approximation: A gradient boosting machine." paper, but uses a few tricks to be much much faster, allowing for better tuning. XGBoost is popular for structured data competitions on Kaggle. Even there: The winner is often an ensemble of XGBoost and Keras. And some structured data competitions are won by neural nets alone: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st-place-interview/ http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it... and https://www.kaggle.com/c/higgs-boson/discussion/10425 https://www.kaggle.com/c/higgs-boson/discussion/10425 (Neural nets won the Higgs Boson Detection Challenge, where XGBoost was introduced) I'd say Tensorflow/Keras can handle a wider variety of problems, with the same, or improved accuracy, than tree-based methods can. NN's do well on structured problems (the domain of tree-based methods), but also own computer vision and, increasingly, NLP. I agree on the pitfalls of applying neural nets vs. forests. It is true that tree-based methods are academically a bit out of vogue: The exciting stuff is happening in the neural network space. You have more chance of getting published with deep learning (this used to be the other way around).

throw_away_777 10y ago
I agree that neural nets are state-of-the-art and do quite well on certain types of problems (NLP and vision, which are important problems). But a lot of data is structured (sales, churn, recommendations, etc), and it is so much easier to train an xgboost model than a neural net model. You need a very expensive computer or expensive cloud computing to train neural nets, and even then it is not easy. Ease of implementation is an important factor that gets overlooked in academia. And on non-NLP and non-image datasets, usually the single best Kaggle model is an xgboost model, which was probably developed in 1/10th the time it took to make a good neural net model. Xgboost has come a long way since it was first introduced, with early stopping being an example of a significant improvement.
- photoJ 10y ago
  How can you say that ease of implementation is overlooked in academia, when academia created the exact tools your are speaking of?

sgt101 10y ago

Because in the real world we need robust classifiers not optimised ones. Problem : robust && optimized are very vague terms in ML.