11 ms·
> Perhaps, but we do know a lot more about how (deep) NNs work than we did 40 years ago. That really depends on what you mean by "a lot more". I have no qualms
by pron 9y ago
> Perhaps, but we do know a lot more about how (deep) NNs work than we did 40 years ago.
That really depends on what you mean by "a lot more". I have no qualms about the ongoing research and steady progress in machine learning. My problem is with claims (hinted in the article, as in many other places) that this progress has anything to do with the progress towards AI. AI has been projected to be within 5 years by optimists and 50 years by pessimists for the last 70 years. I don't think any responsible research in machine learning can say that we can now give a narrower range than 50 or 60 (or even 70) years ago.
> Take for example: Generalization in Deep Learning[1] which gives a theoretical grounding of generalization.
So I gave it a quick read. It's certainly about theory, but I would be amazed if anyone called it anywhere near a breakthrough (and it is only marginally about NNs). It is certainly a step towards understanding the statistics of why we see decent generalizations in many scenarios, especially when considerable hand-tuning of the architecture is involved.
> Yes, and DaVinci invented the helicopter, he just failed to get it working.
There's a difference between advancing theory and practice. It wasn't a breakthrough in theory that led to the invention of LSTM, but better hardware and years of tinkering. Again -- this is important and certainly useful, but my point is that we are nowhere near able to proclaim that deep learning is the right path to AI, or that we are anywhere close to achieving it.
- nl 9y agoMy problem is with claims (hinted in the article, as in many other places) that this progress has anything to do with the progress towards AI That's a different claim to the NN complaint you made.
- pron 9y agoBecause progress is hard to measure when you have no goal to compare it against, and therefore no metric. Most disciplines don't try to measure it and largely avoid making grandiose claims [1]. But some in machine learning, for some reason, choose to measure progress in terms of AI, and by that measure I don't think anyone can point to any substantial progress, let alone to major breakthroughs. But even by any other measure, I don't think there has been a major theoretical breakthrough in machine learning in decades, unlike, say, breakthroughs in complexity theory, distributed systems, and other CS disciplines. In any event, marketers can use the term AI to refer to Quicksort for all I care, but I would suggest that machine learning people avoid that term, which is loaded, ill-defined and with a lot of embarrassing historical baggage of failed promises. Instead, they should be pleased that the discipline has finally produced workable practices that are proving useful in some important domains. [1]: Not all, sadly, but machine learning is certainly among the worst offenders when it comes to claims vs. reality, although programming language theory is occasionally a close contender.
- p1esk 9y agoWhat would be an example of substantial progress/major breakthrough in AI?
- pron 9y agoIn terms of practice, I would say a program that displays the learning and reasoning abilities of some advanced invertebrates (say wasps or spiders). But we don't know how far away from AI that would put us. Once we achieve that, are we 5 years away from human-level intelligence or 50? We simply do not know. In terms of theory, a breakthrough would be a better understanding of what intelligence is on one hand, and how "unorganized systems", to use Turing's terminology, evolve sophisticated algorithms. At some stage, Turing believed that as a precursor to intelligence, we should study simpler biological phenomena, and turned to so-called "artificial life". There hasn't been too much progress on that front, as well, but the work done by Stuart Kauffman [1] since the late sixties seems like moving in the right direction, albeit very slowly. Don't get me wrong: I'm not an AI skeptic. I believe that we will achieve it one day. I just think it is very irresponsible for machine learning researchers to hint we're getting close, when, in fact, they have no idea whether we are or we aren't. To be more specific, we don't know whether deep learning, i.e. deep neural networks trained through a variant of backpropagation is the approach that would one day lead us to AI. [1]: https://en.wikipedia.org/wiki/Stuart_Kauffman https://en.wikipedia.org/wiki/Stuart_Kauffman
- p1esk 9y agoThanks. Do you follow Numenta's work? They published several papers recently (e.g. [1]) [1] https://numenta.com/papers/a-theory-of-how-columns-in-the-neocortex-enable-learning-the-structure-of-the-world/ https://numenta.com/papers/a-theory-of-how-columns-in-the-ne...
- pron 9y agoI find it very hard to judge papers by Numenta, because they make it very hard to separate science from marketing; their entire marketing moto is "we're better because we're more like the actual brain" (their HTM networks), rather than "because we perform drastically better". So claims to better biological accuracy (a controversial direction since the birth of AI) are pretty much their raison d'etre. So they say they're writing about theory in this paper, but I see mostly observations, so it's theory more in the sense of a hypothesis rather than what is meant by "theory" in math or physics. But I am really not qualified to judge this paper's importance.