4 ms·

>"Outside of neural networks, GPUs don’t play a large role in machine learning today, and much larger gains in speed can often be achieved by a careful choice o

>"Outside of neural networks, GPUs don’t play a large role in machine learning today, and much larger gains in speed can often be achieved by a careful choice of algorithms." [1] Scikit learn espouses a non-GPU approach. Perhaps the performance gains by using GPUs aren't that significant. Has anyone tried SVMs (or for that matter other non-DL classifiers) + GPUs? [1] http://scikit-learn.org/stable/faq.html http://scikit-learn.org/stable/faq.html

sanxiyn 9y ago
XGBoost has GPU support. It is recent: the paper was published in 2017. http://xgboost.readthedocs.io/en/latest/gpu/ http://xgboost.readthedocs.io/en/latest/gpu/
- nlperguiy 9y ago
  https://github.com/Microsoft/LightGBM https://github.com/Microsoft/LightGBM LightGBM also has GPU support for learning decision trees.
  - sanxiyn 9y ago
    Note: LightGBM paper https://arxiv.org/abs/1706.08359 https://arxiv.org/abs/1706.08359 cites XGBoost paper (reference 16).
cosmic_ape 9y ago
there is a link to a white paper on the github. they show there is a x10 or more decrease in train time on a couple of datasets.
lz400 9y ago
Anyone has more insight into why? if I try to grid search hyperparameters for a random forest takes ages in a single machine with scikit-learn. I only ever see GPUs in neural networks. Is there some acceleration for non-NN machine learning algos?
- bmh100 9y ago
  As an aside, you should generally avoid grid search in favor of random search or an evolutionary algorithm. Grid search can waste your time budget on likely bad combinations of hyperparameters instead of looking for combinations local to or nearby a known-good combination.
- T-A 9y ago
  Generally speaking, what GPUs excel at is applying exactly the same operation to many parallel data streams. Neural networks are like that. Random forests are the opposite: branching on conditional tests is very expensive. It's been several years since I last wrote raw CUDA and OpenCL, but if memory serves, back then the docs essentially said that every if...else amounted to running both branches on all the data and then deciding what to keep, effectively halving performance. So a decision tree just a few levels deep would slow you down by an order of magnitude.

sanxiyn 9y ago

XGBoost has GPU support. It is recent: the paper was published in 2017. http://xgboost.readthedocs.io/en/latest/gpu/ http://xgboost.readthedocs.io/en/latest/gpu/

nlperguiy 9y ago
https://github.com/Microsoft/LightGBM https://github.com/Microsoft/LightGBM LightGBM also has GPU support for learning decision trees.
- sanxiyn 9y ago
  Note: LightGBM paper https://arxiv.org/abs/1706.08359 https://arxiv.org/abs/1706.08359 cites XGBoost paper (reference 16).

cosmic_ape 9y ago

there is a link to a white paper on the github. they show there is a x10 or more decrease in train time on a couple of datasets.

lz400 9y ago

Anyone has more insight into why? if I try to grid search hyperparameters for a random forest takes ages in a single machine with scikit-learn. I only ever see GPUs in neural networks. Is there some acceleration for non-NN machine learning algos?

bmh100 9y ago
As an aside, you should generally avoid grid search in favor of random search or an evolutionary algorithm. Grid search can waste your time budget on likely bad combinations of hyperparameters instead of looking for combinations local to or nearby a known-good combination.
T-A 9y ago
Generally speaking, what GPUs excel at is applying exactly the same operation to many parallel data streams. Neural networks are like that. Random forests are the opposite: branching on conditional tests is very expensive. It's been several years since I last wrote raw CUDA and OpenCL, but if memory serves, back then the docs essentially said that every if...else amounted to running both branches on all the data and then deciding what to keep, effectively halving performance. So a decision tree just a few levels deep would slow you down by an order of magnitude.