5 ms·
So this ("NN's were invented a long time ago and there has been no theoretical progress since") is a common meme put forward by people who read bits about AI, u
by nl 9y ago
So this ("NN's were invented a long time ago and there has been no theoretical progress since") is a common meme put forward by people who read bits about AI, usually work in software but aren't actively doing machine learning.
It's not true.
For example, the LSTM (the default architecture for use of a NN on text) wasn't invented until 1997.
If we want to look at just very recent work, ResNet wasn't invented until 2015, Attention until 2010 (being generous), NMTs in 2014 etc etc. One can argue these built on other things - which is absolute true - but we don't make the same argument about say Wayland under Linux: "Oh it's just a graphics API with X and Y co-ordinates and more powerful computers".
Until ~2014, people couldn't train neural networks over 4 layers deep. Until ResNet they couldn't do more than ~10 layers. Now hundreds of layers normal for image recognition tasks, and people still say "there is nothing new".
It wasn't faster hardware that got these techniques to work, it was new theoretical breakthroughs.
- pron 9y ago> usually work in software but aren't actively doing machine learning. Well, maybe not actively any more, but I was working with NNs a lot in the 90s. > For example, the LSTM (the default architecture for use of a NN on text) wasn't invented until 1997. 1. We know no more about the theory of LSTM networks than we do about ordinary NNs. 2. McCulloch and Pitts original 1943 paper explicitly mentions hooking a recurrent NN to external memory (they talk about a Turing machine tape). > Until ~2014, people couldn't train neural networks over 4 layers deep. Until ResNet they couldn't do more than ~10 layers. Now hundreds of layers normal for image recognition tasks, and people still say "there is nothing new". That's not what I said. I said that there has been great progress thanks to better hardware and heuristics that have accumulated over the years. But the theory has hardly moved an inch in at least a couple of decades, and has yet to make a breakthrough. > it was new theoretical breakthroughs. AFAIK, it's been all hardware and heuristics (like LSTM) gained through trial an error, but I'd love to see an example of a theoretical breakthrough.
- nl 9y agoWe know no more about the theory of LSTM networks than we do about ordinary NNs. Perhaps, but we do know a lot more about how (deep) NNs work than we did 40 years ago. Take for example: Generalization in Deep Learning[1] which gives a theoretical grounding of generalization. McCulloch and Pitts original 1943 paper explicitly mentions hooking a recurrent NN to external memory (they talk about a Turing machine tape). Yes, and DaVinci invented the helicopter, he just failed to get it working. [1] https://arxiv.org/pdf/1710.05468.pdf https://arxiv.org/pdf/1710.05468.pdf
- pron 9y ago> Perhaps, but we do know a lot more about how (deep) NNs work than we did 40 years ago. That really depends on what you mean by "a lot more". I have no qualms about the ongoing research and steady progress in machine learning. My problem is with claims (hinted in the article, as in many other places) that this progress has anything to do with the progress towards AI. AI has been projected to be within 5 years by optimists and 50 years by pessimists for the last 70 years. I don't think any responsible research in machine learning can say that we can now give a narrower range than 50 or 60 (or even 70) years ago. > Take for example: Generalization in Deep Learning[1] which gives a theoretical grounding of generalization. So I gave it a quick read. It's certainly about theory, but I would be amazed if anyone called it anywhere near a breakthrough (and it is only marginally about NNs). It is certainly a step towards understanding the statistics of why we see decent generalizations in many scenarios, especially when considerable hand-tuning of the architecture is involved. > Yes, and DaVinci invented the helicopter, he just failed to get it working. There's a difference between advancing theory and practice. It wasn't a breakthrough in theory that led to the invention of LSTM, but better hardware and years of tinkering. Again -- this is important and certainly useful, but my point is that we are nowhere near able to proclaim that deep learning is the right path to AI, or that we are anywhere close to achieving it.
- nl 9y agoMy problem is with claims (hinted in the article, as in many other places) that this progress has anything to do with the progress towards AI That's a different claim to the NN complaint you made.