8 ms·

When not to use deep learning

254 points by jpn 9y ago

peterburkimsher 9y ago
My naïve understanding of deep learning is that it works by finding patterns in the answers, instead of actually solving problems. If I take a multiple-choice exam and always answer "C", then I have a good chance at getting more than 25%. For image recognition, I think the classifier is doing the real work (trying to actually answer the question), and the deep learning is just seeing if the answer matches the pattern of expected answers. Somehow, this actually works. I think that it's because true randomness is hard to find. The problem that I've found is that it's really difficult to teach deep learning. I'm making a Chinese-English teaching tool ( http://pingtype.github.io http://pingtype.github.io ) and sourcing my translations from Google Translate. I find a lot of mistakes in my dictionary that obviously came from Google's model getting the word spacing wrong. I can fix it in my own dictionary immediately. If I submit the correction to Google, it just changes some weightings, and hundreds of people will have to submit the same correction before their deep learning will finally catch on that it needs to change something.
- jorgemf 9y ago
  Your brain also works finding patterns. The only difference is that your brain works for a lot of domains and can translate information of one domain to other. We have not find a good way to do this in deep learning yet. But once we find it, it will know that answer C all the time probably is not correct.
- shadowmint 9y ago
  Don't just down vote this guys, that doesn't help. No, your naive understanding is not correct. 'Deep learning', and by that I mean a neural network, works at a super high level by generalising some input (say X) into some related output (lets say, Y). It's not just random choice; it's like defining a programming function: foo(x) -> y { ... } Where the '...' is implemented by a set of statistical weights and training data and so on. ...but ultimately, you raise a valid point. Training models is incredibly time consuming, and incorporating 'corrections' as new training data is extremely non trivial. It's one of the issues with deep learning. I would even go as far as to say, 'You need to regularly update your training data with new examples and counter examples' as a 'when not to use deep learning'.
  - aklemm 9y ago
    There was no mention of random choice (was there?) and the naive description seems to line up with your description.
- sdrothrock 9y ago
  > I can fix it in my own dictionary immediately. If I submit the correction to Google, it just changes some weightings, and hundreds of people will have to submit the same correction before their deep learning will finally catch on that it needs to change something. This isn't a problem with deep learning, but a problem with verifying a data source. You are trusting yourself to submit a proper correction, but Google doesn't know you from John. To use another example, imagine that someone says 他的 is "they're", another person says it's "their", and still another person says it's "there". Your approach would just accept all three in a row, wouldn't it? Google's approach presumably trusts that the crowd + what it sees on the web is correct and thus attempts to verify that a submission is typical language use before actually putting it into play.
- nneonneo 9y ago
  I just tried your tool, and I must say it generated some impressive nonsense. Putting in 这是什么东西 ("What is this thing") in the Chinese box yielded the translation "this is Why tender east west", which is a _character for character_ translation of the phrase. That's not especially meaningful or useful given that Chinese words are polysyllabic - I hope this is a temporary bug. By contrast, Google Translate gives me the much more sensible "What is this" as a translation.
  - peterburkimsher 9y ago
    Thank you for trying it - the reason is that you entered simplified characters, but the tool is for traditional. You can click Advanced -> Regional -> Simplified to Traditional. Entering 這是什麼東西 gives "this is what thing", which has correct word spacing.
- xapata 9y ago
  Finding patterns in answers is solving problems.
  - _0ffh 9y ago
    Maybe more precisely finding patterns in answers in relation to the inputs.
  - freyir 9y ago
    But it doesn't show its work.
  - TheOtherHobbes 9y ago
    No, it's not. Humans seem to solve problems with a combination of learned stock knowledge, induction, and constrained improvisation. Constrained improvisation is the most interesting part of that process, and the one we know least about. It's one thing building a system that asymptotically improves over millions of trials, and then sending out a press release claiming your system is as smart as a human. It's another building a system that learns a domain as efficiently as a human. Compare the relatively small number of games played/analysed by a Go master on their way to master status, compared with the number of simulated games played/analysed by AlphaGo. ML is still a rather naive form of constrained brute forcing. It's a long way short of efficient learning.
    - xapata 9y ago
      You're describing a technique for finding patterns. There are many possible techniques and machines can (probably) use any of them. The constraint would be if the technique involved kinetic or quantum mechanics.
    - digitalzombie 9y ago
      > ML is still a rather naive form of constrained brute forcing. It's a long way short of efficient learning. Uh... Bayesian can do the above at in term of expert domain. They call it elicitation in the Bayesian world. I think you're overall generalizing all learning techniques. And also it's not like we actually really know how the human brain learn. Psychology is a field with huge uncertainties and you can see that in their research papers with correlation values. So the concept of learning may be out dated and/or we are still learning about what makes us learn.
- unityByFreedom 9y ago
  I have the same question about Google translate. I'm not convinced that its errors can simply be chalked up to neural networks. As you said, they ought to be constantly trying to improve their word segmentation and datasets. I'm not convinced that Google Translate is as major of a focus at Google as other things. Perhaps I'm wrong, and it's just a hard problem, but the translations I've seen haven't improved as much over the years as I expected given progress in other areas of AI. They may be more focused on adding new languages than improving existing ones. Don't know. Japanese translation is pretty good. Chinese is bad. I guess Chinese is harder because nobody really uses any phonetic writing for reading beyond grade schools. And then there is traditional/simplified, and, I imagine, other differences between how different regions use characters. Baidu's translate is better than Facebook/Google in some cases. I wonder if translation just isn't as sexy as image recognition and self driving cars, therefore research dollars aren't as focused on it.
  - dgacmu 9y ago
    You're wrong. It gets a ton of focus. Here's the largest update in years (from 10 months ago): https://research.googleblog.com/2016/09/a-neural-network-for-machine.html?m=1 https://research.googleblog.com/2016/09/a-neural-network-for... It's a very hard problem. There are a ton of people at Google, baidu, and elsewhere working on it. (Source: I'm a part timer on Google Brain)
    - unityByFreedom 9y ago
      Do you think with more research dollars it could be improved sooner?
  - PeterisP 9y ago
    Machine translation has a lot of research funding, and has had it for decades. I'm not sure about USA, but it's been a major focus of very large EU grant programs due to the obvious multilinguality of EU and the explicit goal to move towards a single European market by reducing barriers in trade, including language barrier. It also has many commercial use cases and thus has always had quite a lot of people and teams working on it compared to other fields of ML or NLP. The problem is that it's hard. Every 0.1% of progress has historically required a lot of work.
- deepnet 9y ago
  > My naïve understanding of deep learning is that it works by finding patterns in the answers, instead of actually solving problems. I think this is quite profound and inspiring. Although perhaps it is only one half of it. Concretely, deep learning finds patterns, the best patterns are derived from the highest bandwidth signal, often this is the input. Geoff Hinton has argued that the task, solving the problem, is a low bandwidth signal. Hinton aphorises [approximately] 'If you want to learn computer vision first learn to do computer graphics, i.e. a generative model.' - this is about the bandwidth of the data signal. Hinton: [1] "Each image has much more information in it than a typical label... Each image puts a lot of constraint on the identity function. Whereas if I give you an image and a label and I try and get the right answer I don't get much constraint on the mapping from image to label. The bits of constraint on that mapping imposed by training example are just the number of bits to say what the answer is which is not very many." [1] @5:28 http://videolectures.net/mlss09uk_hinton_dbn/ http://videolectures.net/mlss09uk_hinton_dbn/
- enriquto 9y ago
  > My naïve understanding of deep learning is that it works by finding patterns in the answers, instead of actually solving problems. Wow, I love this sentence! Plato: A man is a featherless biped Diogenes (plucking a chicken): Here's your man, Plato! Plato (deep learning): A man is a flat-nailed featherless biped
  - peterburkimsher 9y ago
    Is it a bird? Is it a plane? No, it's Superman! Superman is an alien.
- daddyo 9y ago
  Your naive understanding is supported by at least one deep learning authority: > I haven’t found a way to properly articulate this yet but somehow everything we do in deep learning is memorization (interpolation, pattern recognition, etc) instead of thinking (extrapolation, induction, etc). I haven’t seen a single compelling example of a neural network that I would say “thinks”, in a very abstract and hard-to-define feeling of what properties that would have and what that would look like. > All the while I'm thinking: this thinking process this person goes through as he analyzes this data: THAT is what Machine Learning SHOULD do -- Andrej Karpathy Deep learning for image recognition works because our visual world is made up of structured hierarchical features: Dark/Light, Texture, Edge, Part of Object, Object, Scene. Deep learning layers create increasingly higher-level features in a computationally feasible way.
  - amelius 9y ago
    So a better name for "deep learning" would be "shallow understanding"?
    - lightbyte 9y ago
      Maybe something like subtle memorization?
      - 0xdeadbeefbabe 9y ago
        It's obfuscated memorization though. Otherwise wouldn't youtube search on my "smart" TV yield videos that contain the search term? I searched for "face detection" and got "face recognition" videos. I felt like a linear model would have been more useful.
    - dmreedy 9y ago
      I personally prefer 'generic hashing/parsing'; deep learning excels at the automatic creation of a mapping of unstructured information to structured information, after a sufficient period of training.
      - 9y ago
- nl 9y ago
  Finding patterns in answers isn't unique to deep learning. All machine learning training methods rely on this at some level.
therajiv 9y ago
The author discusses how linear models are generally more interpretable than deep learning methods, but I'd argue that's actually changing pretty quickly. Especially for large image/sequence inputs (which covers most of the applications that are getting hyped up), linear regressions don't perform very well, and often that performance difference prevents them from picking out important features. Given that fast, scalable methods for feature importance are on the rise (e.g. https://arxiv.org/abs/1704.02685 https://arxiv.org/abs/1704.02685, which the author mentions), you often get equally interpretable feature scores from deep models that are more accurate than analogous ones from linear models. Basically, my point is that model interpretation strongly depends on how accurate your model is, and because deep learning models are so much better than linear models for some tasks, it makes sense to use them - even if your primary goal is interpretability. That said, I do believe that if you ever care at all about interpretation, you should almost never be using multilayer perceptrons (which have recently become part of the widening umbrella term "deep learning"), because they rarely work better than decision tree models or basic linear models (and MLPs are generally less or equally as interpretable when compared to traditional methods).
- reality_hacker 9y ago
  But what is the situation in real life? Can I get some feature importance scores say from tensorflow model?
  - therajiv 9y ago
    I'll agree with you that it's much harder than it should be (thankfully, finding the implementations is the hard part, not using them), but yes, these methods do exist. DeepLIFT (the method I linked in my original comment: https://github.com/kundajelab/deeplift https://github.com/kundajelab/deeplift), takes a Keras model (with Theano or TensorFlow backend) as input and provides feature importance scores for any desired layer of the network (raw data inputs, inputs to dense layers following convolution, etc.). Keras-Vis (https://github.com/raghakot/keras-vis https://github.com/raghakot/keras-vis) is another nice package that allows for easy visualization of saliency maps and convolutional filters. Perturbing inputs and looking at the effect on the output of the network is another technique people use pretty often. I think there's a lot of room for this space to become easier to use, especially for newer deep learning practitioners. To that point, I definitely agree with the author of this blog post.
nilkn 9y ago
I was expecting more discussion of alternatives. For instance, in cases where deep neural networks aren't desirable or don't outperform classical approaches, I'm a big fan of boosted decision trees, due to their accuracy on many real-world datasets, their ease-of-use, and the existence of great open source implementations. xgboost (which routinely wins Kaggle competitions) and Spark MLLib both have high-performance distributed training algorithms for gradient boosted trees. And as far as hyperparameter searches go, there just aren't as many parameters to optimize. (And frameworks like Spark are already fantastic for embarrassingly parallel tasks like hyperparameter searches.)
- nl 9y ago
  Spark MLLib both have high-performance distributed training algorithms for gradient boosted trees. Well it exists, but I wouldn't describe it as high performance in either accuracy or speed. I'm a big fan of Spark, but Spark ML needs some love from people who actually use it. Until that happens, just use XGB (which now has Spark integration[1]) [1] http://dmlc.ml/2016/03/14/xgboost4j-portable-distributed-xgboost-in-spark-flink-and-dataflow.html http://dmlc.ml/2016/03/14/xgboost4j-portable-distributed-xgb...
fnl 9y ago
Pretty much agree, and particularly on the budget/time aspect.
digitalzombie 9y ago
If NN was able to do small data then are they better than their counter parts? I mean if you can do it for small data and it was good then we would be seeing it dominate kaggle in all problem domains. Maybe the small data problems belong to other algorithm (such as tree base and forest, SVM). disclaimer - I'm bias for tree base algorithm in medium and small data since it is my thesis.
- andreyk 9y ago
  I think no - SVMs are explicitly optimized to generalize the best from small data (https://en.wikipedia.org/wiki/Hinge_loss https://en.wikipedia.org/wiki/Hinge_loss - 'The hinge loss is used for "maximum-margin" classification'), whereas NNs have more hacky regularization methods. I am not sure if the same is true for tree-based methods, but of course those are lovely due to how interpretable they are when you have few features.
andreyk 9y ago
"The point is that training deep nets carries a big cost, in both computational and debugging time. Such expense doesn’t make sense for lots of day-to-day prediction problems and the ROI of tweaking a deep net to them, even when tweaking small networks, might be too low. " As a Masters student now training deep models for a little while now, I think this point is underemphasized. Doing something novel (so, not just image classification) requires a TON of engineering, not to mention the research considerations. And there are so many tiny decisions and hyperparameters, that even when I thought I had considerable domain knowledge I found it very lacking. I guess it should not be surprising given that 'Deep Learning' refers to a very broad set of models only related by having a learned hierarchical representation. There are a few problems where you can use existing deep learning almost off the shelf (most notably image classification, segmentation), but for most applications I think we're not there yet. As long as this remains true (which I suspect will be for a long time), SVMs and decision trees and linear models are still definitely worth knowing and understanding.

254 points by jpn 9y ago

My naïve understanding of deep learning is that it works by finding patterns in the answers, instead of actually solving problems. If I take a multiple-choice exam and always answer "C", then I have a good chance at getting more than 25%. For image recognition, I think the classifier is doing the real work (trying to actually answer the question), and the deep learning is just seeing if the answer matches the pattern of expected answers. Somehow, this actually works. I think that it's because true randomness is hard to find. The problem that I've found is that it's really difficult to teach deep learning. I'm making a Chinese-English teaching tool ( http://pingtype.github.io http://pingtype.github.io ) and sourcing my translations from Google Translate. I find a lot of mistakes in my dictionary that obviously came from Google's model getting the word spacing wrong. I can fix it in my own dictionary immediately. If I submit the correction to Google, it just changes some weightings, and hundreds of people will have to submit the same correction before their deep learning will finally catch on that it needs to change something.

jorgemf 9y ago
Your brain also works finding patterns. The only difference is that your brain works for a lot of domains and can translate information of one domain to other. We have not find a good way to do this in deep learning yet. But once we find it, it will know that answer C all the time probably is not correct.
shadowmint 9y ago
Don't just down vote this guys, that doesn't help. No, your naive understanding is not correct. 'Deep learning', and by that I mean a neural network, works at a super high level by generalising some input (say X) into some related output (lets say, Y). It's not just random choice; it's like defining a programming function: foo(x) -> y { ... } Where the '...' is implemented by a set of statistical weights and training data and so on. ...but ultimately, you raise a valid point. Training models is incredibly time consuming, and incorporating 'corrections' as new training data is extremely non trivial. It's one of the issues with deep learning. I would even go as far as to say, 'You need to regularly update your training data with new examples and counter examples' as a 'when not to use deep learning'.
- aklemm 9y ago
  There was no mention of random choice (was there?) and the naive description seems to line up with your description.
sdrothrock 9y ago
> I can fix it in my own dictionary immediately. If I submit the correction to Google, it just changes some weightings, and hundreds of people will have to submit the same correction before their deep learning will finally catch on that it needs to change something. This isn't a problem with deep learning, but a problem with verifying a data source. You are trusting yourself to submit a proper correction, but Google doesn't know you from John. To use another example, imagine that someone says 他的 is "they're", another person says it's "their", and still another person says it's "there". Your approach would just accept all three in a row, wouldn't it? Google's approach presumably trusts that the crowd + what it sees on the web is correct and thus attempts to verify that a submission is typical language use before actually putting it into play.
nneonneo 9y ago
I just tried your tool, and I must say it generated some impressive nonsense. Putting in 这是什么东西 ("What is this thing") in the Chinese box yielded the translation "this is Why tender east west", which is a _character for character_ translation of the phrase. That's not especially meaningful or useful given that Chinese words are polysyllabic - I hope this is a temporary bug. By contrast, Google Translate gives me the much more sensible "What is this" as a translation.
- peterburkimsher 9y ago
  Thank you for trying it - the reason is that you entered simplified characters, but the tool is for traditional. You can click Advanced -> Regional -> Simplified to Traditional. Entering 這是什麼東西 gives "this is what thing", which has correct word spacing.
xapata 9y ago
Finding patterns in answers is solving problems.
- _0ffh 9y ago
  Maybe more precisely finding patterns in answers in relation to the inputs.
- freyir 9y ago
  But it doesn't show its work.
- TheOtherHobbes 9y ago
  No, it's not. Humans seem to solve problems with a combination of learned stock knowledge, induction, and constrained improvisation. Constrained improvisation is the most interesting part of that process, and the one we know least about. It's one thing building a system that asymptotically improves over millions of trials, and then sending out a press release claiming your system is as smart as a human. It's another building a system that learns a domain as efficiently as a human. Compare the relatively small number of games played/analysed by a Go master on their way to master status, compared with the number of simulated games played/analysed by AlphaGo. ML is still a rather naive form of constrained brute forcing. It's a long way short of efficient learning.
  - xapata 9y ago
    You're describing a technique for finding patterns. There are many possible techniques and machines can (probably) use any of them. The constraint would be if the technique involved kinetic or quantum mechanics.
  - digitalzombie 9y ago
    > ML is still a rather naive form of constrained brute forcing. It's a long way short of efficient learning. Uh... Bayesian can do the above at in term of expert domain. They call it elicitation in the Bayesian world. I think you're overall generalizing all learning techniques. And also it's not like we actually really know how the human brain learn. Psychology is a field with huge uncertainties and you can see that in their research papers with correlation values. So the concept of learning may be out dated and/or we are still learning about what makes us learn.
unityByFreedom 9y ago
I have the same question about Google translate. I'm not convinced that its errors can simply be chalked up to neural networks. As you said, they ought to be constantly trying to improve their word segmentation and datasets. I'm not convinced that Google Translate is as major of a focus at Google as other things. Perhaps I'm wrong, and it's just a hard problem, but the translations I've seen haven't improved as much over the years as I expected given progress in other areas of AI. They may be more focused on adding new languages than improving existing ones. Don't know. Japanese translation is pretty good. Chinese is bad. I guess Chinese is harder because nobody really uses any phonetic writing for reading beyond grade schools. And then there is traditional/simplified, and, I imagine, other differences between how different regions use characters. Baidu's translate is better than Facebook/Google in some cases. I wonder if translation just isn't as sexy as image recognition and self driving cars, therefore research dollars aren't as focused on it.
- dgacmu 9y ago
  You're wrong. It gets a ton of focus. Here's the largest update in years (from 10 months ago): https://research.googleblog.com/2016/09/a-neural-network-for-machine.html?m=1 https://research.googleblog.com/2016/09/a-neural-network-for... It's a very hard problem. There are a ton of people at Google, baidu, and elsewhere working on it. (Source: I'm a part timer on Google Brain)
  - unityByFreedom 9y ago
    Do you think with more research dollars it could be improved sooner?
- PeterisP 9y ago
  Machine translation has a lot of research funding, and has had it for decades. I'm not sure about USA, but it's been a major focus of very large EU grant programs due to the obvious multilinguality of EU and the explicit goal to move towards a single European market by reducing barriers in trade, including language barrier. It also has many commercial use cases and thus has always had quite a lot of people and teams working on it compared to other fields of ML or NLP. The problem is that it's hard. Every 0.1% of progress has historically required a lot of work.
deepnet 9y ago
> My naïve understanding of deep learning is that it works by finding patterns in the answers, instead of actually solving problems. I think this is quite profound and inspiring. Although perhaps it is only one half of it. Concretely, deep learning finds patterns, the best patterns are derived from the highest bandwidth signal, often this is the input. Geoff Hinton has argued that the task, solving the problem, is a low bandwidth signal. Hinton aphorises [approximately] 'If you want to learn computer vision first learn to do computer graphics, i.e. a generative model.' - this is about the bandwidth of the data signal. Hinton: [1] "Each image has much more information in it than a typical label... Each image puts a lot of constraint on the identity function. Whereas if I give you an image and a label and I try and get the right answer I don't get much constraint on the mapping from image to label. The bits of constraint on that mapping imposed by training example are just the number of bits to say what the answer is which is not very many." [1] @5:28 http://videolectures.net/mlss09uk_hinton_dbn/ http://videolectures.net/mlss09uk_hinton_dbn/
enriquto 9y ago
> My naïve understanding of deep learning is that it works by finding patterns in the answers, instead of actually solving problems. Wow, I love this sentence! Plato: A man is a featherless biped Diogenes (plucking a chicken): Here's your man, Plato! Plato (deep learning): A man is a flat-nailed featherless biped
- peterburkimsher 9y ago
  Is it a bird? Is it a plane? No, it's Superman! Superman is an alien.
daddyo 9y ago
Your naive understanding is supported by at least one deep learning authority: > I haven’t found a way to properly articulate this yet but somehow everything we do in deep learning is memorization (interpolation, pattern recognition, etc) instead of thinking (extrapolation, induction, etc). I haven’t seen a single compelling example of a neural network that I would say “thinks”, in a very abstract and hard-to-define feeling of what properties that would have and what that would look like. > All the while I'm thinking: this thinking process this person goes through as he analyzes this data: THAT is what Machine Learning SHOULD do -- Andrej Karpathy Deep learning for image recognition works because our visual world is made up of structured hierarchical features: Dark/Light, Texture, Edge, Part of Object, Object, Scene. Deep learning layers create increasingly higher-level features in a computationally feasible way.
- amelius 9y ago
  So a better name for "deep learning" would be "shallow understanding"?
  - lightbyte 9y ago
    Maybe something like subtle memorization?
    - 0xdeadbeefbabe 9y ago
      It's obfuscated memorization though. Otherwise wouldn't youtube search on my "smart" TV yield videos that contain the search term? I searched for "face detection" and got "face recognition" videos. I felt like a linear model would have been more useful.
  - dmreedy 9y ago
    I personally prefer 'generic hashing/parsing'; deep learning excels at the automatic creation of a mapping of unstructured information to structured information, after a sufficient period of training.
    - 9y ago
nl 9y ago
Finding patterns in answers isn't unique to deep learning. All machine learning training methods rely on this at some level.

therajiv 9y ago

The author discusses how linear models are generally more interpretable than deep learning methods, but I'd argue that's actually changing pretty quickly. Especially for large image/sequence inputs (which covers most of the applications that are getting hyped up), linear regressions don't perform very well, and often that performance difference prevents them from picking out important features. Given that fast, scalable methods for feature importance are on the rise (e.g. https://arxiv.org/abs/1704.02685 https://arxiv.org/abs/1704.02685, which the author mentions), you often get equally interpretable feature scores from deep models that are more accurate than analogous ones from linear models. Basically, my point is that model interpretation strongly depends on how accurate your model is, and because deep learning models are so much better than linear models for some tasks, it makes sense to use them - even if your primary goal is interpretability. That said, I do believe that if you ever care at all about interpretation, you should almost never be using multilayer perceptrons (which have recently become part of the widening umbrella term "deep learning"), because they rarely work better than decision tree models or basic linear models (and MLPs are generally less or equally as interpretable when compared to traditional methods).

reality_hacker 9y ago
But what is the situation in real life? Can I get some feature importance scores say from tensorflow model?
- therajiv 9y ago
  I'll agree with you that it's much harder than it should be (thankfully, finding the implementations is the hard part, not using them), but yes, these methods do exist. DeepLIFT (the method I linked in my original comment: https://github.com/kundajelab/deeplift https://github.com/kundajelab/deeplift), takes a Keras model (with Theano or TensorFlow backend) as input and provides feature importance scores for any desired layer of the network (raw data inputs, inputs to dense layers following convolution, etc.). Keras-Vis (https://github.com/raghakot/keras-vis https://github.com/raghakot/keras-vis) is another nice package that allows for easy visualization of saliency maps and convolutional filters. Perturbing inputs and looking at the effect on the output of the network is another technique people use pretty often. I think there's a lot of room for this space to become easier to use, especially for newer deep learning practitioners. To that point, I definitely agree with the author of this blog post.

nilkn 9y ago

I was expecting more discussion of alternatives. For instance, in cases where deep neural networks aren't desirable or don't outperform classical approaches, I'm a big fan of boosted decision trees, due to their accuracy on many real-world datasets, their ease-of-use, and the existence of great open source implementations. xgboost (which routinely wins Kaggle competitions) and Spark MLLib both have high-performance distributed training algorithms for gradient boosted trees. And as far as hyperparameter searches go, there just aren't as many parameters to optimize. (And frameworks like Spark are already fantastic for embarrassingly parallel tasks like hyperparameter searches.)

nl 9y ago
Spark MLLib both have high-performance distributed training algorithms for gradient boosted trees. Well it exists, but I wouldn't describe it as high performance in either accuracy or speed. I'm a big fan of Spark, but Spark ML needs some love from people who actually use it. Until that happens, just use XGB (which now has Spark integration[1]) [1] http://dmlc.ml/2016/03/14/xgboost4j-portable-distributed-xgboost-in-spark-flink-and-dataflow.html http://dmlc.ml/2016/03/14/xgboost4j-portable-distributed-xgb...

fnl 9y ago

Pretty much agree, and particularly on the budget/time aspect.

digitalzombie 9y ago

If NN was able to do small data then are they better than their counter parts? I mean if you can do it for small data and it was good then we would be seeing it dominate kaggle in all problem domains. Maybe the small data problems belong to other algorithm (such as tree base and forest, SVM). disclaimer - I'm bias for tree base algorithm in medium and small data since it is my thesis.

andreyk 9y ago
I think no - SVMs are explicitly optimized to generalize the best from small data (https://en.wikipedia.org/wiki/Hinge_loss https://en.wikipedia.org/wiki/Hinge_loss - 'The hinge loss is used for "maximum-margin" classification'), whereas NNs have more hacky regularization methods. I am not sure if the same is true for tree-based methods, but of course those are lovely due to how interpretable they are when you have few features.

andreyk 9y ago

"The point is that training deep nets carries a big cost, in both computational and debugging time. Such expense doesn’t make sense for lots of day-to-day prediction problems and the ROI of tweaking a deep net to them, even when tweaking small networks, might be too low. " As a Masters student now training deep models for a little while now, I think this point is underemphasized. Doing something novel (so, not just image classification) requires a TON of engineering, not to mention the research considerations. And there are so many tiny decisions and hyperparameters, that even when I thought I had considerable domain knowledge I found it very lacking. I guess it should not be surprising given that 'Deep Learning' refers to a very broad set of models only related by having a learned hierarchical representation. There are a few problems where you can use existing deep learning almost off the shelf (most notably image classification, segmentation), but for most applications I think we're not there yet. As long as this remains true (which I suspect will be for a long time), SVMs and decision trees and linear models are still definitely worth knowing and understanding.