9 ms·
The problem I see with this article is that it starts with problems with deep learning today and extrapolates to say that researchers won't get past them: "Non
by skybrian 9y ago
The problem I see with this article is that it starts with problems with deep learning today and extrapolates to say that researchers won't get past them:
"None of these shortcomings is likely to be solved soon."
There's no evidence for this, and I think it underestimates the large and well-funded machine learning community. Maybe they'll run into a brick wall, but who's to say a bunch of smart researchers won't figure out ways around today's limitations?
Without physical constraints or impossibility results, I think the only rational outside view is that this is unpredictable. For any of today's pressing problems with machine learning, maybe someone will post a new research paper tomorrow with a different approach that solves it. Or maybe not?
- randcraw 9y agoThe 50 year history of AI research portends the difficulty in surmounting secondary obstacles unearthed by each new AI technique's innovations. The limits of deep learning are almost certainly no different from past statistically-driven methods in reaching comparable limits, especially those obstacles that show no signs of vulnerability to the slings and arrows inherent in rich semantics. The recent article by Doug Hofstadter in The Atlantic on using DNNs to do machine translation highlights this nicely. To date, no work in DNNs has shown any potential for redressing problems of the common sense knowledge needed to know when your translated sentence is semantic nonsense. Likewise, no statistic-based learning method has shown itself capable of tapping into much less creating the deep knowledgebases and wealth of relations that give bare facts the semantic meaning needed to convey or understand subtleties in messages that are more than trivial. Deft mimicry may win a Turing Test, but complex syntheses of thought (like writing a fictional story in which characters have internal thoughts and hidden motives that depend on relationships, subtexts, and dependencies) — no form of AI has yet shown any potential to solve such problems. Too much internal state and complex relations must be modeled, much less, the ability to extend and translate these to many possible worlds — as humans do every day. No, statistics-based AI techniques (like deep nets) show little promise to grow indefinitely unto full (wo)manhood.
- skybrian 9y agoI'm just an outside observer, but I wouldn't say "no potential". For example, I'm reminded of a paper [1] about a technique that allows translation between language pairs that weren't in the training data. This doesn't rise to the level of common sense yet, but it seems impressive, and there is a language-independent portion of the neural net that seems to be encoding something. [1] https://research.googleblog.com/2016/11/zero-shot-translation-with-googles.html https://research.googleblog.com/2016/11/zero-shot-translatio...
- YeGoblynQueenne 9y ago>>> "None of these shortcomings is likely to be solved soon." >> There's no evidence for this, and I think it underestimates the large and well-funded machine learning community. Well- deep learning has been around for a few decades now. LeCun's LeNet-5 is already 20 years old. LSTM is 21 (from the Hochreiter and Schmidhuber paper). Those are the two best-known today architectures, but work on deep nets goes way back to the '80s. So those are old algorithms and techniques and they've had the same limitations for 20+ years. What makes them "likely to be solved soon" now?
- skybrian 9y agoA lot more attention. The field seems to be moving pretty fast and doesn't seem to be stuck. But I wouldn't say "likely" or "unlikely" but rather that it's uncertain.
- YeGoblynQueenne 9y agoTwo problems with that. First, what determines whether a problem is close to solving is usually not the amount of interest in it but, rather, its hardness. See for instance P = NP, or the drive to find a cure for "cancer". Neither has been solved yet despite considerable effort and investment (particularly in cancer cures). The limitations of deep learning are a similarly hard problem- they're "baked in" to its DNA so it's a very hard problem how to overcome them [1]. Second, yep- the field is stuck, well and truly stuck. The fact that LSTM and CNN are 20+ years old is telling. The vast majority of ANN research today is basically trying small tweaks to old ideas to see if something will come out of them. There is very little incentive to research something radically different- because people can already make a very lucrative career with little timid architectural tweaks and researching something new and untried is risky. So even if just throwing a lot of money and minds to a hard problem could just make it go away, nobody is really trying. It will be a long time before a majority of researchers recognise the limitations of deep learning and in general statistical machine learning and do something about them. ___________________ [1] The article links to an article by François Chollet of Keras. That's the second part of two. The first one is orders of magnitude better informed than the article above and recommended background reading in any discussion about the limitations of deep learning: https://blog.keras.io/the-limitations-of-deep-learning.html https://blog.keras.io/the-limitations-of-deep-learning.html