6 ms·
A 'Brief' History of Neural Nets and Deep Learning
- mturmon 11y agoUpvoted -- what a nice and detailed history. Thanks for linking to the old NYT article on Frank Rosenblatt's work. One can see how researchers of the time were irked by delirious press releases when the credit-assignment problem for multilayer nets had not been addressed. (We managed to mostly address the credit-assignment problem for multilayer nets...but the delirious press release problem remains unsolved.) Incidentally, it's "Seymour Papert", not "Paper" (appears twice).
- andreyk 11y agoThank you, I will correct the typo.
- MrQuincle 11y agoThis is too long ago for me to know what actually happened. Both Rosenblatt as well as Minsky and Papert knew that networks could perfectly learn the XOR function. See https://en.wikipedia.org/wiki/Perceptrons_(book) https://en.wikipedia.org/wiki/Perceptrons_(book) and also stated in this article. I knew there was a fight between Minsky and Grossberg at that time. Perhaps there have been other reasons that are not so well known that led to an AI winter. Have these winters ever be quantified though? For the article, you can find it in here: http://msrvideo.vo.msecnd.net/rmcvideos/258318/dl/258318.pdf http://msrvideo.vo.msecnd.net/rmcvideos/258318/dl/258318.pdf
- BasDirks 11y agoThis is actually a very important chapter in human development. And it will pass in the blink of an eye. Works like these are good, important.
- FreedomToCreate 11y agoGreat read. Its incredible what we, as a species have been able to achieve in the last 2 centuries. I feel like we are where scientists were with computers in the 1950s. We are starting to see the big picture, but its wide application is still decades away.
- dahart 11y agoExcellent article, thanks for the contribution. Perhaps some independent validation, but I was coincidentally having this conversation the other day with a relatively well known computer vision researcher, about why it seems like the idea of neural nets has floundered for decades and suddenly it's the hot topic, and we're seeing massively improved results. His answers, summarized, are that: 1- Big data is making possible the kind of training we could never do before. 2- Having big data & big compute has made some training breakthroughs that allowed the depth to increase dramatically. The number of layers was implicitly limited until recently because anything deep couldn't be practically trained. 3- The activation function has very commonly in the past been an S-curve, and some of the newer better results are using a linear function that is clamped on the low end at zero, but not clamped on top. All really interesting to me. This is making me want to implement and play with neural nets! Of course, now the big question: if we have a neural net big enough, and it works, can we simulate a human brain? (Apparently, according to my AI researcher friend, we're not there yet with the foundational building blocks. He mentioned researchers have tried simulating a life-form known to have a small number of neurons, like a thousand, and they can't get it to work yet.)
- dwiel 11y agoI agree with everything but the last. Simulating a human brain is perhaps necessary for consciousness uploading, but not for super human ai. We have been able to travel faster by machine than foot for a long time, in spite of not being able to build a robot that can simulate human running.
- dahart 11y ago> Simulating a human brain is perhaps necessary for consciousness uploading, but not for super human ai. I'm not suggesting otherwise, and what you said might be true, I don't know. But honest question: how can we know it's true before we can do either? How can we suggest neural nets are a good building block of super human AI, if we can't verify we can do regular human thinking, or sub-human thinking? Because the neural net is modeled after (simplified) brain neurons in the first place, it seems reasonable to suggest that if we can't simulate a simple brain, then we can't validate that the model is correct, right? It might be very productive, and it might 'work' in some sense, but we don't know whether it can truly act as the lego brick of brain building material until we can build a functional brain. Your observation is true, and worth considering. Machines can travel faster than humans, but that's different in part because we didn't start trying to simulate the human foot, right? The computational equivalent would be that machines can multiply a lot of numbers much faster than a human. That is super-human calculation, and it is mechanical thinking, of a sort, but most people wouldn't say it counts as "AI", perhaps the same way that most people wouldn't say a car counts as human running, even if it is faster. Neural nets are being used to classify images, find objects, identify people's faces in a crowd or in difficult to see situations. But, they currently can't tell you to stop classifying images because you're asking the wrong question, or interrupt you to say you're looking great today.
- ausjke 11y agoGreat article, took an neural network course while I was doing my graduate study long time ago and it might be time to resume that subject. the NN training then took a long time to be impractical for real use and now it should be much faster.
- signa11 11y agojust to re-iterate, the reference text #13 I.e. parallel distributed processing (vol. 1&2) are also an excellent introduction to the field, from its infancy. contains a collection of papers by nn luminaries including rumelhart, Hinton etc. Very highly recommended.
- lovelearning 11y agoAuthor's done an excellent job of explaining what the problems were at every stage and how NNs evolved to solve them. Learnt a lot from this series.