4 ms·

Yet another piece that cherrypicks AI failures in order to make vague thesis that "deep learning is not enough". I can't decide if it makes the article more or

Yet another piece that cherrypicks AI failures in order to make vague thesis that "deep learning is not enough". I can't decide if it makes the article more or less honest that the "Beyond Deep Learning" section is essentially a catalog of... recent developments in Deep Learning. The need to (selectively) pidgeonhole and then judge (on impossible to operationalize criteria) a fast-evolving field is purely rhetorical and, worse, aimed at folks who don't understand the way the field is progressing. I've got nothing against raising platforms for alternative lines of AI research, especially including highlighting the ways they complement current deep learning But this article format is so transparent and tired.

eli_gottlieb 8y ago
If you actually read the papers they cite, you'll notice they bring in probabilistic programming as well as other alternatives to deep learning.
- evrydayhustling 8y ago
  I didn't say they didn't! Yes, they "bring in" probabilistic programming, graphical representations of symbolic structures, and more. But all of the citations 6-10 are from papers that integrate new approaches into a deep neural model. Most are from self- described deep learning teams, and all would be welcome at conferences and publications within the deep learning community. Integrating with other inference paradigms has been a major driver in neural network research for 30y... so trying to cast progress like this as some sort of comeuppance that proves the insufficiency of the field is BS.
  - eli_gottlieb 8y ago
    >But all of the citations 6-10 are from papers that integrate new approaches into a deep neural model. Having read citation 8, "Building Machines that Learn and Think like People", a number of times over, I don't see where that paper includes a neural network architecture.
    - sh33mp 8y ago
      The above commenter probably spoke a little too hastily. "Building Machines" is indeed not a paper about a neural network method, but a survey of problems that we should expect neural networks to do better on, but they currently aren't. That said, the paper isn't down on deep learning, but rather says "we need extra stuff, either additional methods or more inductive biases in our models".
    - evrydayhustling 8y ago
      Your correct - that one doesn't propose an architecture but suggests that others do so. It also is arguably from "outside" the main deep learning community, and in the context of the OP could be interpreted as corrective. However, that is exactly not the paper's own stance, which is explicitly constructive to the field. And it is reciprocated! Check out all the NeurIPS papers citing and relating their work to the suggestions here: https://scholar.google.com/scholar?cites=8504723689348856287&as_sdt=5,33&sciodt=0,33&hl=en https://scholar.google.com/scholar?cites=8504723689348856287... While I misspoke, my point remains that both this contribution and it's reception are part of a healthy community, not a case of "moving past" or exposing shortcomings of a field..
      - eli_gottlieb 8y ago
        >While I misspoke, my point remains that both this contribution and it's reception are part of a healthy community, not a case of "moving past" or exposing shortcomings of a field.. From within "the field", there isn't a single "deep learning community" counterposed to an "other stuff community". There's a broad machine learning community, in which deep learning is currently the most fashionable approach, and there are small factions of pro- and anti-"deep" partisans, who consider their approach the One True Approach. Science is usually unkind to partisans, and so most people I've met don't really bother with that kind of partisanship. Asking, "Can deep learning get us all the way to AGI, yes or no?" is the wrong question. The proper questions are: * What is deep learning good or bad at? * What do we mean by AGI? (Similarly: data analytics, machine perception, statistical AI, etc?) * Why should we target one goal (high-dimensional statistics, missing-data reconstruction, reinforcement learning, AGI) over another? * What methods currently afford us the best contributions towards the scientific and engineering goals we're targeting?
dmreedy 8y ago
The criteria certainly are impossible to judge currently. Which makes your assertion that the field is 'progressing' strange. I think there's a lot of merit to remembering the last time we got this excited about a given model for AI. And the time before that. And the time before that. Go back further, to the time that we were all so excited that we could model the entire world with a little bit of beautiful logic. And the subsequent disappointment. There is so much nonsense out there, so much hand-waving, mixed signals, false promises, downright naivety, and both malicious and completely inadvertent miscommunication. A little rigor in a field this excitable would be nice, I say.
- YeGoblynQueenne 8y ago
  Unfortunately, when rigor crawls in, excitement flies out. Rigorous machine learning would be really boring for everyone except its most dedicated followers. It's probably how Geoff Hinton and his students soldiered on with their research through alternating waves of lay hype and disappointment.
- RivieraKid 8y ago
  > Which makes your assertion that the field is 'progressing' strange. Well, it is progressing. That assertion is correct.
- evrydayhustling 8y ago
  The progress is well defined. Both academic and industrial deep learning groups aggressively define challenge tasks in vision, NLP, relational learning and more, then document progress on them. There is even an unprecedented release of code and datasets to let others replicate and extend work. Detractions like your "so much nonsense out there" pick straw man goalposts - whether it's AGI (whatever that really means) or "the craziest thing I heard from IBM marketing". The fact that some people are talking about crazy milestones doesn't mean that there aren't real ones being met.
RivieraKid 8y ago
Yeah, I started typing a comment trying to say something like that but was unable to put it into words. I've only skimmed through the article but in general, these criticisms of DL are annoying, they rarely bring new or valuable information. It's usually a mix of disproving claims nobody is making (e.g. that DL is close to general AI) or stating something that's obvious.

eli_gottlieb 8y ago

If you actually read the papers they cite, you'll notice they bring in probabilistic programming as well as other alternatives to deep learning.

evrydayhustling 8y ago
I didn't say they didn't! Yes, they "bring in" probabilistic programming, graphical representations of symbolic structures, and more. But all of the citations 6-10 are from papers that integrate new approaches into a deep neural model. Most are from self- described deep learning teams, and all would be welcome at conferences and publications within the deep learning community. Integrating with other inference paradigms has been a major driver in neural network research for 30y... so trying to cast progress like this as some sort of comeuppance that proves the insufficiency of the field is BS.
- eli_gottlieb 8y ago
  >But all of the citations 6-10 are from papers that integrate new approaches into a deep neural model. Having read citation 8, "Building Machines that Learn and Think like People", a number of times over, I don't see where that paper includes a neural network architecture.
  - sh33mp 8y ago
    The above commenter probably spoke a little too hastily. "Building Machines" is indeed not a paper about a neural network method, but a survey of problems that we should expect neural networks to do better on, but they currently aren't. That said, the paper isn't down on deep learning, but rather says "we need extra stuff, either additional methods or more inductive biases in our models".
  - evrydayhustling 8y ago
    Your correct - that one doesn't propose an architecture but suggests that others do so. It also is arguably from "outside" the main deep learning community, and in the context of the OP could be interpreted as corrective. However, that is exactly not the paper's own stance, which is explicitly constructive to the field. And it is reciprocated! Check out all the NeurIPS papers citing and relating their work to the suggestions here: https://scholar.google.com/scholar?cites=8504723689348856287&as_sdt=5,33&sciodt=0,33&hl=en https://scholar.google.com/scholar?cites=8504723689348856287... While I misspoke, my point remains that both this contribution and it's reception are part of a healthy community, not a case of "moving past" or exposing shortcomings of a field..
    - eli_gottlieb 8y ago
      >While I misspoke, my point remains that both this contribution and it's reception are part of a healthy community, not a case of "moving past" or exposing shortcomings of a field.. From within "the field", there isn't a single "deep learning community" counterposed to an "other stuff community". There's a broad machine learning community, in which deep learning is currently the most fashionable approach, and there are small factions of pro- and anti-"deep" partisans, who consider their approach the One True Approach. Science is usually unkind to partisans, and so most people I've met don't really bother with that kind of partisanship. Asking, "Can deep learning get us all the way to AGI, yes or no?" is the wrong question. The proper questions are: * What is deep learning good or bad at? * What do we mean by AGI? (Similarly: data analytics, machine perception, statistical AI, etc?) * Why should we target one goal (high-dimensional statistics, missing-data reconstruction, reinforcement learning, AGI) over another? * What methods currently afford us the best contributions towards the scientific and engineering goals we're targeting?

dmreedy 8y ago

The criteria certainly are impossible to judge currently. Which makes your assertion that the field is 'progressing' strange. I think there's a lot of merit to remembering the last time we got this excited about a given model for AI. And the time before that. And the time before that. Go back further, to the time that we were all so excited that we could model the entire world with a little bit of beautiful logic. And the subsequent disappointment. There is so much nonsense out there, so much hand-waving, mixed signals, false promises, downright naivety, and both malicious and completely inadvertent miscommunication. A little rigor in a field this excitable would be nice, I say.

YeGoblynQueenne 8y ago
Unfortunately, when rigor crawls in, excitement flies out. Rigorous machine learning would be really boring for everyone except its most dedicated followers. It's probably how Geoff Hinton and his students soldiered on with their research through alternating waves of lay hype and disappointment.
RivieraKid 8y ago
> Which makes your assertion that the field is 'progressing' strange. Well, it is progressing. That assertion is correct.
evrydayhustling 8y ago
The progress is well defined. Both academic and industrial deep learning groups aggressively define challenge tasks in vision, NLP, relational learning and more, then document progress on them. There is even an unprecedented release of code and datasets to let others replicate and extend work. Detractions like your "so much nonsense out there" pick straw man goalposts - whether it's AGI (whatever that really means) or "the craziest thing I heard from IBM marketing". The fact that some people are talking about crazy milestones doesn't mean that there aren't real ones being met.

RivieraKid 8y ago

Yeah, I started typing a comment trying to say something like that but was unable to put it into words. I've only skimmed through the article but in general, these criticisms of DL are annoying, they rarely bring new or valuable information. It's usually a mix of disproving claims nobody is making (e.g. that DL is close to general AI) or stating something that's obvious.