4 ms·

I'm going to go against the grain here and (gasp) not hate on deep learning. People should realize that although many older statistical methods and "traditional

I'm going to go against the grain here and (gasp) not hate on deep learning. People should realize that although many older statistical methods and "traditional" machine learning methods such as LDA, SVMs and decision trees may be good enough for business tasks, they are not the cutting edge of AI research. I think people are forgetting exactly how difficult image classification and object detection was before the advent of deep learning. People hating on "stack more layers" forget that "stack more layers" is EXACTLY what improved imagenet performance to such as massive extent. ResNets pushed the limits by figuring out how to stack more layers in a beneficial way. And let's take a look at AlphaGo, how would you do that with SVMs or decision trees? Just get over the fact that deep learning provides a level of "intuition" (Go's search space is famously greater than the estimated number of stars in the universe). I think that a part of the problem is that older ML PhDs are angry that deep learning is so easy (until the learning rate fails to provide convergence of course...) and would prefer that their preferred methods would still reign supreme. I'll end this wall of text on the note that OpenAI's Ian Goodfellow notes that all projects at OpenAI use deep learning right now, but they are not dogmatic and will consider other approaches if they work well. I think this is the path that should be taken. On the other hand, I also see a bright future for uniting traditional techniques with deep learning such as attaching a SVM to a CNN and combining decision trees with CNNs both of which have resulted in good results.

dmreedy 10y ago
>> I think that a part of the problem is that older ML PhDs are angry that deep learning is so easy (until the learning rate fails to provide convergence of course...) and would prefer that their preferred methods would still reign supreme. I think that's definitely a part of it, and I feel that way sometimes myself (not that I'm a PhD). But there's another side of that reluctance that lies on the axis of model accountability and explicability. A lot of modern ML/Deep Learning doesn't -feel- like we're understanding anything any more than we did ten years ago. Yes our black-box results are better according to the tests we've laid out for them, but there's something more slippery about the 'why', beyond the handwave of 'complexity'. Maybe this is just the way it will be going forward (in the spirit of Quantum's "shut up and calculate"), but it is not easy to give up something that you can wrap your head around with something that kind of just takes care of itself, especially if you're in the business of seeking knowledge instead of results.
- deepnotderp 10y ago
  I'm not sure I fully buy the "explainability" argument against deep learning since our goal is human level intelligence and human intelligence isn't easily explained either. On the other hand, we're nowhere near human level intelligence in most tasks (Go, Poker and image classificaiton non withholding) so I can understand the argument for explainability from a practical perspective. I think we're making some good progress in that direction though, and I'll list them below: 1) Attention maps in CNNs can tell us what the net is usually looking at. 2) "Attentive Explanations" use attention mechanisms to point to the object of interest to generate an explanation for VQA tasks, check the paper (warning PDF): https://arxiv.org/pdf/1612.04757 https://arxiv.org/pdf/1612.04757 3) A recent project used a similar explanation mechanism that forced the network to output "what it was thinking" while playing an Atari game. 4) NTMs (neural turing machines) allow weighted memory access, which alleviates the back box issue to some extent.
  - dmreedy 10y ago
    >> since our goal is human level intelligence and human intelligence isn't easily explained either I think this might be where the real divide lies. To present a false dichotomy, there are two schools. One that is interested in building systems that attain human-level intelligence, and another that is interested in understanding the nature of human intelligence by building models using our closes understood analog (computation). We're definitely making progress on Deep Learning models that tell us more about how they think they work, but philosophically, I'm not sure we're making progress on understanding the nature of what they experience. I don't mean this in any polemical sense, just trying to explain the particular dissatisfaction (and it -is- dissatisfaction, not contempt or disdain. The stuff is amazing) I feel towards these advances.
  - romaniv 10y ago
    >I'm not sure I fully buy the "explainability" argument against deep learning since our goal is human level intelligence and human intelligence isn't easily explained either. There are two types of explainability. One is explaining why the system converges on correct solutions in general. Another is explaining why a particular decision was made the way it was made. Human intelligence is usually very much explainable in the second sense. This is not always true, but such cases are called "intuition" and we tends not to trust such judgements for mission-critical tasks.
feral 10y ago
You can argue against Cuban's point that anyone who doesn't know machine learning will be "dinosaur in 3 years", while also being enthusiastic about deep learning. > "traditional" machine learning methods ... may be good enough for business tasks, they are not the cutting edge of AI research That's true - but so? Under discussion is whether every developer should drop what they are doing and learn ML/DL, not what the cutting edge of AI research is? In that context a lot of the low-hanging fruit for the next while is going to involve actually deploying simpler ML, with all the non-ML work that entails - and maybe using off-the-shelf DL to extract information from unstructured data. > older ML PhDs are angry that deep learning is so easy That's very ungenerous. People tend to do PhDs in things like ML (often making sacrifices to do so) because they want to advance the field - not to make it harder! People are excited. They are wary about hype causing another AI winter, or a lot of important projects failing for silly reasons.
- deepnotderp 10y ago
  I agree that Cuban is being foolish here, I was just talking about the technical perspective.
romaniv 10y ago
>And let's take a look at AlphaGo, how would you do that with SVMs or decision trees? This is a poor example. There are Go programs that work very well and use exclusively MCTS without any neural networks.
- deepnotderp 10y ago
  This is a fallacy, AlphaGo didn't increase very much with additional hardware at all, it's elo would still have been higher than lee se-dol's elo. And yes, MCTS worked well for Go, but notice how the combination boosted it to new levels? And deep nets without MCTS also worked very well on Go, outperforming MCTS only approaches (by FAIR and was the top before being dethroned by AlphaGo).
  - romaniv 10y ago
    I edited the post because I thought someone will nitpick on my comment about hardware, but it seems you caught the older version. > This is a fallacy, AlphaGo didn't increase very much with additional hardware at all It did. Please take a look at the page 36 of the original paper. (https://gogameguru.com/i/2016/03/deepmind-mastering-go.pdf https://gogameguru.com/i/2016/03/deepmind-mastering-go.pdf) The performance did not grow linearly, but the increases were vert significant. >it's elo would still have been higher than lee se-dol's elo I seriously doubt it would win that match. One-GPU version lost to 8-GPU version around 95% of all games. The paper is somewhat misleading in estimating relative performance of AlphaGo compared to older programs, because (as far as I know) while all of them were run on the same hardware, only AlphaGo used GPU acceleration.
deleted 10y ago
[deleted]
reachtarunhere 10y ago
Agree on the idea of Classical Machine Learning experts being a little pissed regarding the average joe getting better results with such ease. A lot of these people have put in years of study/work in those techniques and it must be sad to see some form of sudden paradigm shift making it worth less. On the other hand it is true that DL is not required for many (most?) companies as they don't work with the kind of problems you list (vision, robotics)

dmreedy 10y ago

>> I think that a part of the problem is that older ML PhDs are angry that deep learning is so easy (until the learning rate fails to provide convergence of course...) and would prefer that their preferred methods would still reign supreme. I think that's definitely a part of it, and I feel that way sometimes myself (not that I'm a PhD). But there's another side of that reluctance that lies on the axis of model accountability and explicability. A lot of modern ML/Deep Learning doesn't -feel- like we're understanding anything any more than we did ten years ago. Yes our black-box results are better according to the tests we've laid out for them, but there's something more slippery about the 'why', beyond the handwave of 'complexity'. Maybe this is just the way it will be going forward (in the spirit of Quantum's "shut up and calculate"), but it is not easy to give up something that you can wrap your head around with something that kind of just takes care of itself, especially if you're in the business of seeking knowledge instead of results.

deepnotderp 10y ago
I'm not sure I fully buy the "explainability" argument against deep learning since our goal is human level intelligence and human intelligence isn't easily explained either. On the other hand, we're nowhere near human level intelligence in most tasks (Go, Poker and image classificaiton non withholding) so I can understand the argument for explainability from a practical perspective. I think we're making some good progress in that direction though, and I'll list them below: 1) Attention maps in CNNs can tell us what the net is usually looking at. 2) "Attentive Explanations" use attention mechanisms to point to the object of interest to generate an explanation for VQA tasks, check the paper (warning PDF): https://arxiv.org/pdf/1612.04757 https://arxiv.org/pdf/1612.04757 3) A recent project used a similar explanation mechanism that forced the network to output "what it was thinking" while playing an Atari game. 4) NTMs (neural turing machines) allow weighted memory access, which alleviates the back box issue to some extent.
- dmreedy 10y ago
  >> since our goal is human level intelligence and human intelligence isn't easily explained either I think this might be where the real divide lies. To present a false dichotomy, there are two schools. One that is interested in building systems that attain human-level intelligence, and another that is interested in understanding the nature of human intelligence by building models using our closes understood analog (computation). We're definitely making progress on Deep Learning models that tell us more about how they think they work, but philosophically, I'm not sure we're making progress on understanding the nature of what they experience. I don't mean this in any polemical sense, just trying to explain the particular dissatisfaction (and it -is- dissatisfaction, not contempt or disdain. The stuff is amazing) I feel towards these advances.
- romaniv 10y ago
  >I'm not sure I fully buy the "explainability" argument against deep learning since our goal is human level intelligence and human intelligence isn't easily explained either. There are two types of explainability. One is explaining why the system converges on correct solutions in general. Another is explaining why a particular decision was made the way it was made. Human intelligence is usually very much explainable in the second sense. This is not always true, but such cases are called "intuition" and we tends not to trust such judgements for mission-critical tasks.

feral 10y ago

You can argue against Cuban's point that anyone who doesn't know machine learning will be "dinosaur in 3 years", while also being enthusiastic about deep learning. > "traditional" machine learning methods ... may be good enough for business tasks, they are not the cutting edge of AI research That's true - but so? Under discussion is whether every developer should drop what they are doing and learn ML/DL, not what the cutting edge of AI research is? In that context a lot of the low-hanging fruit for the next while is going to involve actually deploying simpler ML, with all the non-ML work that entails - and maybe using off-the-shelf DL to extract information from unstructured data. > older ML PhDs are angry that deep learning is so easy That's very ungenerous. People tend to do PhDs in things like ML (often making sacrifices to do so) because they want to advance the field - not to make it harder! People are excited. They are wary about hype causing another AI winter, or a lot of important projects failing for silly reasons.

deepnotderp 10y ago
I agree that Cuban is being foolish here, I was just talking about the technical perspective.

romaniv 10y ago

>And let's take a look at AlphaGo, how would you do that with SVMs or decision trees? This is a poor example. There are Go programs that work very well and use exclusively MCTS without any neural networks.

deepnotderp 10y ago
This is a fallacy, AlphaGo didn't increase very much with additional hardware at all, it's elo would still have been higher than lee se-dol's elo. And yes, MCTS worked well for Go, but notice how the combination boosted it to new levels? And deep nets without MCTS also worked very well on Go, outperforming MCTS only approaches (by FAIR and was the top before being dethroned by AlphaGo).
- romaniv 10y ago
  I edited the post because I thought someone will nitpick on my comment about hardware, but it seems you caught the older version. > This is a fallacy, AlphaGo didn't increase very much with additional hardware at all It did. Please take a look at the page 36 of the original paper. (https://gogameguru.com/i/2016/03/deepmind-mastering-go.pdf https://gogameguru.com/i/2016/03/deepmind-mastering-go.pdf) The performance did not grow linearly, but the increases were vert significant. >it's elo would still have been higher than lee se-dol's elo I seriously doubt it would win that match. One-GPU version lost to 8-GPU version around 95% of all games. The paper is somewhat misleading in estimating relative performance of AlphaGo compared to older programs, because (as far as I know) while all of them were run on the same hardware, only AlphaGo used GPU acceleration.

deleted 10y ago

[deleted]

reachtarunhere 10y ago

Agree on the idea of Classical Machine Learning experts being a little pissed regarding the average joe getting better results with such ease. A lot of these people have put in years of study/work in those techniques and it must be sad to see some form of sudden paradigm shift making it worth less. On the other hand it is true that DL is not required for many (most?) companies as they don't work with the kind of problems you list (vision, robotics)