6 ms·
What bothers me about articles like this is not their faith in the power of neural networks -- power that, I'm sure, we have only just begun to harness effectiv
by pron 9y ago
What bothers me about articles like this is not their faith in the power of neural networks -- power that, I'm sure, we have only just begun to harness effectively -- but that by ignoring the history of neural networks (and machine learning in general), they paint a somewhat misleading picture.
Artificial neural networks were invented in 1943, some years before the earliest electronic computer prototypes. In fact, the McCulloch & Pitts paper that first described them served as an inspiration (possibly the main inspiration) for von Neumann's design of the digital circuits that would make the earliest computers. Alan Turing discussed using genetic algorithms for training neural networks in 1948[1], roughly concurrently with the earliest work on software. So "software 2.0", if not predated "software 1.0", certainly evolved concurrently with it right from the very beginning.
Early progress seemed so promising that a whole movement dedicated to the belief of the power of neural networks and similar techniques, called Cybernetics, was established in 1948[2] by some of the leading minds of that generation, like Norbert Wiener and von Neumann. They were so optimistic that they believed that a full mathematical theory of intelligence and the brain would be completed in five years. The skeptical Turing, who attended some of the groups meetings but found McCulloch to be a charlatan, thought it would take 50.
80 years later, while better hardware has certainly finally allowed us to enjoy some of the fruits of that old promise, we have yet to make any theoretical breakthrough. The algorithms we use today for training neural networks were invented 50 years ago, and there wasn't even a modest theoretical breakthrough in at least a few decades. I don't think we are in any position to announce any sort of revolution.
[0]: https://en.wikipedia.org/wiki/Artificial_neuron https://en.wikipedia.org/wiki/Artificial_neuron
[1]: https://en.wikipedia.org/wiki/Unorganized_machine https://en.wikipedia.org/wiki/Unorganized_machine
[2]: https://en.wikipedia.org/wiki/Cybernetics https://en.wikipedia.org/wiki/Cybernetics
- monkmartinez 9y agoYou stated that Andrej Karpathy ignores the history of neural networks and posits a misleading picture. While you gave a very brief history of Neural Networks, you failed (imo, BIG TIME) to make a point. First, you really should look into what Mr. Karpathy has done. Second, none of these "old" algorithms were useful in the 40's and 50' because there was no internet and very little data. They are only becoming useful now due to the explosion of data. So, it stands to reason that it Mr. Karpathy isn't ignoring history out of malice, he is ignoring it because it doesn't fucking matter... at all.
- pron 9y agoThe point, simply put, is that neural networks have felt like a revolution for 75 years, while little actual progress has been made. That they have finally proven to be effective at statistical clustering, and so are becoming very practical for some important tasks, is very cool, but declaring them to be a revolution or the road to AI is premature. The attitude displayed in the post is not only unjustified by the actual achievement (we are still far from achieving even insect-level "intelligence" or actually replacing a significant portion of "software 1.0") but has actually seriously harmed the very research subject, that I assume Mr. Karpathy wants to foster, in the past. AI research has been seriously burned by over-enthusiastic optimism before; it should learn the lesson.
- hackerfromthefu 9y agoThanks for the deeper perspective, very interesting. Not the OP, but what has happened that is perhaps revolutionary is the hardware capabilities are now unlocking capabilities that were latent in the existing algorithms, but not previously accessible due to hardware limitations. Sometimes quantity creates it's own quality.
- nl 9y agoSo this ("NN's were invented a long time ago and there has been no theoretical progress since") is a common meme put forward by people who read bits about AI, usually work in software but aren't actively doing machine learning. It's not true. For example, the LSTM (the default architecture for use of a NN on text) wasn't invented until 1997. If we want to look at just very recent work, ResNet wasn't invented until 2015, Attention until 2010 (being generous), NMTs in 2014 etc etc. One can argue these built on other things - which is absolute true - but we don't make the same argument about say Wayland under Linux: "Oh it's just a graphics API with X and Y co-ordinates and more powerful computers". Until ~2014, people couldn't train neural networks over 4 layers deep. Until ResNet they couldn't do more than ~10 layers. Now hundreds of layers normal for image recognition tasks, and people still say "there is nothing new". It wasn't faster hardware that got these techniques to work, it was new theoretical breakthroughs.
- pron 9y ago> usually work in software but aren't actively doing machine learning. Well, maybe not actively any more, but I was working with NNs a lot in the 90s. > For example, the LSTM (the default architecture for use of a NN on text) wasn't invented until 1997. 1. We know no more about the theory of LSTM networks than we do about ordinary NNs. 2. McCulloch and Pitts original 1943 paper explicitly mentions hooking a recurrent NN to external memory (they talk about a Turing machine tape). > Until ~2014, people couldn't train neural networks over 4 layers deep. Until ResNet they couldn't do more than ~10 layers. Now hundreds of layers normal for image recognition tasks, and people still say "there is nothing new". That's not what I said. I said that there has been great progress thanks to better hardware and heuristics that have accumulated over the years. But the theory has hardly moved an inch in at least a couple of decades, and has yet to make a breakthrough. > it was new theoretical breakthroughs. AFAIK, it's been all hardware and heuristics (like LSTM) gained through trial an error, but I'd love to see an example of a theoretical breakthrough.
- nl 9y agoWe know no more about the theory of LSTM networks than we do about ordinary NNs. Perhaps, but we do know a lot more about how (deep) NNs work than we did 40 years ago. Take for example: Generalization in Deep Learning[1] which gives a theoretical grounding of generalization. McCulloch and Pitts original 1943 paper explicitly mentions hooking a recurrent NN to external memory (they talk about a Turing machine tape). Yes, and DaVinci invented the helicopter, he just failed to get it working. [1] https://arxiv.org/pdf/1710.05468.pdf https://arxiv.org/pdf/1710.05468.pdf