6 ms·

ThunderSVM: A Fast SVM Library on GPUs and CPUs

99 points by lainon 9y ago

deleted 9y ago
[deleted]
AbacusAvenger 9y ago
Potentially stupid question: What is an SVM? The linked page uses the acronym almost 30 times but doesn't define it.
- lainon 9y ago
  Support Vector Machine
- visarga 9y ago
  I know how it feels. SVM's are an old and tried classification method that was in fashion before the DL craze. > An SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall. One advantage of SVM's is that they don't use all data points to decide on the separation plane, just the closest points to the gap (the support vectors), making it more invariant. Another advantage is that they can efficiently perform a non linear classification using the kernel trick, implicitly mapping their inputs into high dimensional feature spaces (here kernel means a distance function between two data points). https://en.wikipedia.org/wiki/Support_vector_machine https://en.wikipedia.org/wiki/Support_vector_machine https://en.wikipedia.org/wiki/Positive-definite_kernel https://en.wikipedia.org/wiki/Positive-definite_kernel I recommend Andrew Ng's ML course for learning about SVM's.
- abhgh 9y ago
  Support Vector Machine. A Machine Learning technique that's used typically for classification and regression, but also has been adapted to novelty detection, stuctured prediction, ranking etc. I wrote up a beginner's tutorial some time back, if you are interested [1]. [1] https://blog.statsbot.co/support-vector-machines-tutorial-c1618e635e93 https://blog.statsbot.co/support-vector-machines-tutorial-c1...
  - roumenguha 9y ago
    Great read, thanks!
- make3 9y ago
  I don't want to be too much of an asshole but don't you have Google like the rest of us. I always think it's so weird when people ask something to the comments in HN of all places when they could literally just have textually written it in Wikipedia or wtv and gotten a better answer
  - xcambar 9y ago
    Congratulations, you made it! You are now, by your own words, too much of an asshole. What an achievement! Please allow me to explain my reasoning: I'm of those with no clue about the meaning of SVM, and I came to the comments section looking for an answer to the question "what the ... is SVM anyway?". Even though I know the way to Google (https://duckduckgo.com https://duckduckgo.com, right?), I'm happy someone asked this question and another answered it and as a consequence saved me (and hopefully others) a couple of clicks and taught me something that I might have otherwise been too lazy to investigate myself. Should I want to be picky, I'd even suggest that having the question answered in the comment section is environmentally more efficient than your suggestion that we should all open a new tab and do the same research. Save a tree, answer the not-so-obvious questions "for the masses".
    - dr_zoidberg 9y ago
      Not only saved a couple of clicks, bu there were some good resources thrown around. When trying to learn a new topic without much knowledge, the most difficult part is finding good reads, videos, tutorials, etc, that will set you on track.
- manojlds 9y ago
  Before I joined the ML team at my company I too had no clue. Now, it is what we talk about 30% of the time :)
bobosha 9y ago
>"Outside of neural networks, GPUs don’t play a large role in machine learning today, and much larger gains in speed can often be achieved by a careful choice of algorithms." [1] Scikit learn espouses a non-GPU approach. Perhaps the performance gains by using GPUs aren't that significant. Has anyone tried SVMs (or for that matter other non-DL classifiers) + GPUs? [1] http://scikit-learn.org/stable/faq.html http://scikit-learn.org/stable/faq.html
- sanxiyn 9y ago
  XGBoost has GPU support. It is recent: the paper was published in 2017. http://xgboost.readthedocs.io/en/latest/gpu/ http://xgboost.readthedocs.io/en/latest/gpu/
  - nlperguiy 9y ago
    https://github.com/Microsoft/LightGBM https://github.com/Microsoft/LightGBM LightGBM also has GPU support for learning decision trees.
    - sanxiyn 9y ago
      Note: LightGBM paper https://arxiv.org/abs/1706.08359 https://arxiv.org/abs/1706.08359 cites XGBoost paper (reference 16).
- cosmic_ape 9y ago
  there is a link to a white paper on the github. they show there is a x10 or more decrease in train time on a couple of datasets.
- lz400 9y ago
  Anyone has more insight into why? if I try to grid search hyperparameters for a random forest takes ages in a single machine with scikit-learn. I only ever see GPUs in neural networks. Is there some acceleration for non-NN machine learning algos?
  - 9y ago
jamilbk 9y ago
Prerequisites - Supported Operating Systems: Linux, Windows and MacOS - CUDA 7.5 or above | cmake 2.8 or above | gcc 4.8 or above s/GPU/nVIDIA/ Is it just me or does anyone else get annoyed when a library claiming to run on GPUs uses proprietary CUDA, making it nvidia-only? It would seem strange to claim that a library "runs on CPUs" but only supports Intel CPUs.
- IanCal 9y ago
  It does run on GPUs, just only some of them. > It would seem strange to claim that a library "runs on CPUs" but only supports Intel CPUs. Most things would "run on CPUs" but often not run on ARM CPUs. The key distinction is that it runs on some GPUs rather than (just) some CPUs, so it seems reasonable that they say that.
  - jamilbk 9y ago
    Most things that "run on CPUs" run on CPUs by many vendors. In the cases where software does not run on ARM processors it's almost always due to an architectural difference with x86, not an artificial proprietary limitation. If an open-source library chose to rely on a proprietary C compiler with special language features (e.g. the Borland C compiler) that targeted only Intel x86 CPUs, I would argue we would not generally claim this software "runs on CPUs". The software "runs on Intel x86 CPUs" seems more appropriate. Similarly, especially in the machine learning field, it seems to be more and more common that "GPU" really means "CUDA-required GPU". In a world where AMD and Intel GPUs are also common (perhaps more common?), to me it makes sense to be up-front about this. I'm not criticizing the authors' choice to use CUDA here. I'm just saying it would be nice if we stop pretending CUDA is synonymous with GPU programming, especially in cases where OpenCL would be a very appropriate choice (e.g. open-source software).
    - IanCal 9y ago
      The important point is it runs on any GPUs. If you think then saying it runs on GPUs is wrong the counter sounds more unusual, that it doesn't run on GPUs. They could have been more precise but there's nothing wrong or particularly misleading about what they said, and what they said conveyed the most important aspect. > would argue we would not generally claim this software "runs on CPUs" "This software does not run on CPUs" sounds wrong for this scenario.

99 points by lainon 9y ago

deleted 9y ago

[deleted]

AbacusAvenger 9y ago

Potentially stupid question: What is an SVM? The linked page uses the acronym almost 30 times but doesn't define it.

lainon 9y ago
Support Vector Machine
visarga 9y ago
I know how it feels. SVM's are an old and tried classification method that was in fashion before the DL craze. > An SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall. One advantage of SVM's is that they don't use all data points to decide on the separation plane, just the closest points to the gap (the support vectors), making it more invariant. Another advantage is that they can efficiently perform a non linear classification using the kernel trick, implicitly mapping their inputs into high dimensional feature spaces (here kernel means a distance function between two data points). https://en.wikipedia.org/wiki/Support_vector_machine https://en.wikipedia.org/wiki/Support_vector_machine https://en.wikipedia.org/wiki/Positive-definite_kernel https://en.wikipedia.org/wiki/Positive-definite_kernel I recommend Andrew Ng's ML course for learning about SVM's.
abhgh 9y ago
Support Vector Machine. A Machine Learning technique that's used typically for classification and regression, but also has been adapted to novelty detection, stuctured prediction, ranking etc. I wrote up a beginner's tutorial some time back, if you are interested [1]. [1] https://blog.statsbot.co/support-vector-machines-tutorial-c1618e635e93 https://blog.statsbot.co/support-vector-machines-tutorial-c1...
- roumenguha 9y ago
  Great read, thanks!
make3 9y ago
I don't want to be too much of an asshole but don't you have Google like the rest of us. I always think it's so weird when people ask something to the comments in HN of all places when they could literally just have textually written it in Wikipedia or wtv and gotten a better answer
- xcambar 9y ago
  Congratulations, you made it! You are now, by your own words, too much of an asshole. What an achievement! Please allow me to explain my reasoning: I'm of those with no clue about the meaning of SVM, and I came to the comments section looking for an answer to the question "what the ... is SVM anyway?". Even though I know the way to Google (https://duckduckgo.com https://duckduckgo.com, right?), I'm happy someone asked this question and another answered it and as a consequence saved me (and hopefully others) a couple of clicks and taught me something that I might have otherwise been too lazy to investigate myself. Should I want to be picky, I'd even suggest that having the question answered in the comment section is environmentally more efficient than your suggestion that we should all open a new tab and do the same research. Save a tree, answer the not-so-obvious questions "for the masses".
  - dr_zoidberg 9y ago
    Not only saved a couple of clicks, bu there were some good resources thrown around. When trying to learn a new topic without much knowledge, the most difficult part is finding good reads, videos, tutorials, etc, that will set you on track.
manojlds 9y ago
Before I joined the ML team at my company I too had no clue. Now, it is what we talk about 30% of the time :)

bobosha 9y ago

>"Outside of neural networks, GPUs don’t play a large role in machine learning today, and much larger gains in speed can often be achieved by a careful choice of algorithms." [1] Scikit learn espouses a non-GPU approach. Perhaps the performance gains by using GPUs aren't that significant. Has anyone tried SVMs (or for that matter other non-DL classifiers) + GPUs? [1] http://scikit-learn.org/stable/faq.html http://scikit-learn.org/stable/faq.html

sanxiyn 9y ago
XGBoost has GPU support. It is recent: the paper was published in 2017. http://xgboost.readthedocs.io/en/latest/gpu/ http://xgboost.readthedocs.io/en/latest/gpu/
- nlperguiy 9y ago
  https://github.com/Microsoft/LightGBM https://github.com/Microsoft/LightGBM LightGBM also has GPU support for learning decision trees.
  - sanxiyn 9y ago
    Note: LightGBM paper https://arxiv.org/abs/1706.08359 https://arxiv.org/abs/1706.08359 cites XGBoost paper (reference 16).
cosmic_ape 9y ago
there is a link to a white paper on the github. they show there is a x10 or more decrease in train time on a couple of datasets.
lz400 9y ago
Anyone has more insight into why? if I try to grid search hyperparameters for a random forest takes ages in a single machine with scikit-learn. I only ever see GPUs in neural networks. Is there some acceleration for non-NN machine learning algos?
- 9y ago

jamilbk 9y ago

Prerequisites - Supported Operating Systems: Linux, Windows and MacOS - CUDA 7.5 or above | cmake 2.8 or above | gcc 4.8 or above s/GPU/nVIDIA/ Is it just me or does anyone else get annoyed when a library claiming to run on GPUs uses proprietary CUDA, making it nvidia-only? It would seem strange to claim that a library "runs on CPUs" but only supports Intel CPUs.

IanCal 9y ago
It does run on GPUs, just only some of them. > It would seem strange to claim that a library "runs on CPUs" but only supports Intel CPUs. Most things would "run on CPUs" but often not run on ARM CPUs. The key distinction is that it runs on some GPUs rather than (just) some CPUs, so it seems reasonable that they say that.
- jamilbk 9y ago
  Most things that "run on CPUs" run on CPUs by many vendors. In the cases where software does not run on ARM processors it's almost always due to an architectural difference with x86, not an artificial proprietary limitation. If an open-source library chose to rely on a proprietary C compiler with special language features (e.g. the Borland C compiler) that targeted only Intel x86 CPUs, I would argue we would not generally claim this software "runs on CPUs". The software "runs on Intel x86 CPUs" seems more appropriate. Similarly, especially in the machine learning field, it seems to be more and more common that "GPU" really means "CUDA-required GPU". In a world where AMD and Intel GPUs are also common (perhaps more common?), to me it makes sense to be up-front about this. I'm not criticizing the authors' choice to use CUDA here. I'm just saying it would be nice if we stop pretending CUDA is synonymous with GPU programming, especially in cases where OpenCL would be a very appropriate choice (e.g. open-source software).
  - IanCal 9y ago
    The important point is it runs on any GPUs. If you think then saying it runs on GPUs is wrong the counter sounds more unusual, that it doesn't run on GPUs. They could have been more precise but there's nothing wrong or particularly misleading about what they said, and what they said conveyed the most important aspect. > would argue we would not generally claim this software "runs on CPUs" "This software does not run on CPUs" sounds wrong for this scenario.