5 ms·

This is true, but only in the academic research world. SVMs had relatively little success on practical problems and in industry, so they never built up the kind

by rm999 9y ago

This is true, but only in the academic research world. SVMs had relatively little success on practical problems and in industry, so they never built up the kind of standing that neural networks did. Even in 2003-2005 - arguably the peak time for SVMs - neural networks were much better known to almost everyone (industry practitioners, researchers, and laypeople) than SVMs. What frustrates me is that people who are starting out in machine learning often never learn that linear/logistic regression dominates the practical applications of ML. I've spoken to people who know the in-and-outs of various deep network architectures who don't even know how to start with building a baseline logistic regression model.

tnecniv 9y ago
What prevented SVMs from catching on in industry?
- rcar 9y ago
  They tend to create difficult to interpret models that don't perform as well as other "black box" modeling methods (GBMs, neural nets, etc.)
  - closed 9y ago
    Was this true, or perceived as true in 2003? My understanding was that people did not see them as performing worse than NN back then.
    - rm999 9y ago
      Definitely true. I worked for a company that was generating millions of dollars a year from neural networks in the mid 90s (edit: to be clear I didn't work there in the 90s, I joined years after their initial buildouts). The Unreasonable Effectiveness™ of neural networks has been true for a long time. When I worked there I tried switching out some models with SVMs and they were less accurate and took 1-2 orders of magnitude more time to train.
      - closed 9y ago
        Really useful to hear, thanks! I know psychology was gaining a lot of headway with NN models in the 90s, but had little sense for what was going on in industry.
  - abhgh 9y ago
    This is not really true. Aside from ensemble models they tend to perform pretty much at par or better. Here's an extensive comparison by Rich Caruana [1] [1] https://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icml06.pdf https://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.ic...
- rm999 9y ago
  the tldr answer: SVMs are theoretically powerful but either impractical or pointless. The longer answer: SVMs comes with two theoretical benefits: 1. A guaranteed optimal solution. You get this with simpler techniques like logistic regression. 2. The ability to use non-linear kernels, which can offer much more power than logistic regression (on par with neural networks). So, at first, it seemed like SVMs were the best of all worlds, and a lot of people got excited. In practice, though, non-linear kernels slowed training down to the point of being impractical. Linear kernels were fast enough, but removed the second benefit, so most people would prefer to use established linear techniques like logistic regression.
- shas3 9y ago
  Also, hyperparameter optimization is a bit painful. Zoubin Ghahramani of Cambridge is a champion for Gaussian process models which are as flexible as SVM/SVC, but have more disciplined approach: via well-chosen priors (Bayesian structure) and single-step hyperparameter optimization.
- tanilama 9y ago
  1.SVM is not interpretable just as DL. 2.Hard to parallel if you're using kernels other than linear one. 3.So-so performance.
- abhgh 9y ago
  Have been in the industry for quite a while, and I think one of the often-overlooked non-technical reasons is they are hard to get an intuitive feel of, compared to ANNs. This might seem irrelevant, but think about it - someone who doesn't have a degree/rigorous training in ML or a ML heavy program (and this population is big in the industry, people who are looking to get into ML, from say analytics or software dev) will try out things he/she can identify with. ANNs are positioned exactly right for this - they're sufficiently sophisticated, you can quickly get a high-level idea, and there is the attractive comparison to how our mind works. So that's what people try out early amongst advanced algos. The industry doesn't give you a lot of time to explore, so once you have invested time to pick up ANNs, you tend to hold on to the knowledge. I've also conducted multiple training sessions/discussions with small groups on ML, and it supports what I've said. SVMs are hard to explain to a general crowd; with ANNs I've enough visual cues to get them started. Sure, they mightn't get the math right away, but they understand enough to be comfortable using a library. As an aside, a comment here mentions Logistic/linear regression dominates the industry. I think it is for a similar reason. They're simple to understand and try out. That doesn't make them good models, in my experience, on a bunch of real-world problems. Now if you ask me about the technical cons of SVMs, I'd say - scalability of non-linear kernels and the fact that I've to cherry-pick kernels. Linear and RBF kernels work well on most problems, but even then, for a bunch of problems where RBF seems to work well, the number of support vectors stored by the model can be massive. If I weren't to be pedantic about it and excuse the fact that the kernel seems to be "memorizing" more than "learning", this is still a beast to run in real time. nu-SVMs address this issue to an extent, but then we are back to picking the right kernel for the task. This is one thing I love about ANNs - the kernel (or what essentially is the kernel) is learned.
YZF 9y ago
Weren't SVMs used for the NetFlix recommendation engine? (i.e. the NetFlix prize?)
- denzil_correa 9y ago
  I think it was Gradient Decision Boosting Trees to combine different models.
- eggie5 9y ago
  No, it was an ensemble of Collaborative filtering using Matrix Factorization (using SVD) and a RBM.
deleted 9y ago
[deleted]

by rm999 9y ago

tnecniv 9y ago

What prevented SVMs from catching on in industry?

rcar 9y ago
They tend to create difficult to interpret models that don't perform as well as other "black box" modeling methods (GBMs, neural nets, etc.)
- closed 9y ago
  Was this true, or perceived as true in 2003? My understanding was that people did not see them as performing worse than NN back then.
  - rm999 9y ago
    Definitely true. I worked for a company that was generating millions of dollars a year from neural networks in the mid 90s (edit: to be clear I didn't work there in the 90s, I joined years after their initial buildouts). The Unreasonable Effectiveness™ of neural networks has been true for a long time. When I worked there I tried switching out some models with SVMs and they were less accurate and took 1-2 orders of magnitude more time to train.
    - closed 9y ago
      Really useful to hear, thanks! I know psychology was gaining a lot of headway with NN models in the 90s, but had little sense for what was going on in industry.
- abhgh 9y ago
  This is not really true. Aside from ensemble models they tend to perform pretty much at par or better. Here's an extensive comparison by Rich Caruana [1] [1] https://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icml06.pdf https://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.ic...
rm999 9y ago
the tldr answer: SVMs are theoretically powerful but either impractical or pointless. The longer answer: SVMs comes with two theoretical benefits: 1. A guaranteed optimal solution. You get this with simpler techniques like logistic regression. 2. The ability to use non-linear kernels, which can offer much more power than logistic regression (on par with neural networks). So, at first, it seemed like SVMs were the best of all worlds, and a lot of people got excited. In practice, though, non-linear kernels slowed training down to the point of being impractical. Linear kernels were fast enough, but removed the second benefit, so most people would prefer to use established linear techniques like logistic regression.
shas3 9y ago
Also, hyperparameter optimization is a bit painful. Zoubin Ghahramani of Cambridge is a champion for Gaussian process models which are as flexible as SVM/SVC, but have more disciplined approach: via well-chosen priors (Bayesian structure) and single-step hyperparameter optimization.
tanilama 9y ago
1.SVM is not interpretable just as DL. 2.Hard to parallel if you're using kernels other than linear one. 3.So-so performance.
abhgh 9y ago
Have been in the industry for quite a while, and I think one of the often-overlooked non-technical reasons is they are hard to get an intuitive feel of, compared to ANNs. This might seem irrelevant, but think about it - someone who doesn't have a degree/rigorous training in ML or a ML heavy program (and this population is big in the industry, people who are looking to get into ML, from say analytics or software dev) will try out things he/she can identify with. ANNs are positioned exactly right for this - they're sufficiently sophisticated, you can quickly get a high-level idea, and there is the attractive comparison to how our mind works. So that's what people try out early amongst advanced algos. The industry doesn't give you a lot of time to explore, so once you have invested time to pick up ANNs, you tend to hold on to the knowledge. I've also conducted multiple training sessions/discussions with small groups on ML, and it supports what I've said. SVMs are hard to explain to a general crowd; with ANNs I've enough visual cues to get them started. Sure, they mightn't get the math right away, but they understand enough to be comfortable using a library. As an aside, a comment here mentions Logistic/linear regression dominates the industry. I think it is for a similar reason. They're simple to understand and try out. That doesn't make them good models, in my experience, on a bunch of real-world problems. Now if you ask me about the technical cons of SVMs, I'd say - scalability of non-linear kernels and the fact that I've to cherry-pick kernels. Linear and RBF kernels work well on most problems, but even then, for a bunch of problems where RBF seems to work well, the number of support vectors stored by the model can be massive. If I weren't to be pedantic about it and excuse the fact that the kernel seems to be "memorizing" more than "learning", this is still a beast to run in real time. nu-SVMs address this issue to an extent, but then we are back to picking the right kernel for the task. This is one thing I love about ANNs - the kernel (or what essentially is the kernel) is learned.

YZF 9y ago

Weren't SVMs used for the NetFlix recommendation engine? (i.e. the NetFlix prize?)

denzil_correa 9y ago
I think it was Gradient Decision Boosting Trees to combine different models.
eggie5 9y ago
No, it was an ensemble of Collaborative filtering using Matrix Factorization (using SVD) and a RBM.

deleted 9y ago

[deleted]