4 ms·
This is an opinionated video that tries to rewrite history. For example, according to it, the "big breakthrough" with deep learning occurred in 2012 when Andre
by cs702 10y ago
This is an opinionated video that tries to rewrite history. For example, according to it, the "big breakthrough" with deep learning occurred in 2012 when Andrew Ng et al got an autoencoder to learn to categorize objects in unlabeled images. WHAT? Many other researchers were doing similar work years earlier. According to whom was this the "big breakthrough?"
The video at least mentions Yann LeCun's early work with convnets, but there's no mention of Hinton et al's work with RBMs and DBNs in 2006, or Bengio et al's work with autoencoders in 2006/2007, or Schmidhuber et al's invention of LSTM cells in 1997... I could keep going. The list of people whose work is insulted by omission is HUGE.
I stopped watching the video at that point.
- raverbashing 10y agoYeah. Between LeCunn and the Google experiment a lot of things happened Not to mention the use of ReLu and understanding of the Vanishing Gradient issue
- pinouchon 10y agoOmitting the other members of the LBH conspiracy and Schmidhuber is unfortunate, but I agree with the idea that the number one reason deep learning is working now is scale. Hinton also says it himself, for example (at 5m45 in the video below): "What was wrong in the 80 was that we didn't have enough data and we didn't have enough compute power. [...] Those were the main things that were wrong". He gives a "brief history of backpropagation" here: https://youtu.be/l2dVjADTEDU?t=4m35s https://youtu.be/l2dVjADTEDU?t=4m35s
- lars 10y agoI agree that scale is an important factor in deep learning's success, but that Google experiment ended up being a good example of how not to do it. They used 16000 CPU cores to get that cat detector. A short while later, a group at Baidu were able to replicate the same network with only 3 computers with 4 GPUs each. (The latter group was also lead by Andrew Ng.)
- espadrine 10y agoIncidentally, seeing the speaker set up an overkill neural network for a trivial classification problem seemed off to me. Unsurprisingly, at least 75% of the neurons were unused. Throwing a phenomenal amount of neurons at a problem is not a goal; using a minimal amount to solve it in a given time budget is. The statement at the end of the video, “all the serious applications from here on out need to have deep learning and AI inside”, seems awfully misguided. Even DeepMind doesn't use deep learning for everything.
- withfries2 10y agocs702, this deck started as a short talk so I didn't have time to acknowledge all the great work leading up to the Google YouTube experiment. Your list is good, and I'd add Rosenblatt for the first perceptron, Rumelhart & McClelland for applying techniques to perception, Werbos for backpropogation, Fukushima for convolutional networks, and so many more. I found these helpful while researching the history: http://www.andreykurenkov.com/writing/a-brief-history-of-neural-nets-and-deep-learning/ http://www.andreykurenkov.com/writing/a-brief-history-of-neu... http://www.scholarpedia.org/article/Deep_Learning http://www.scholarpedia.org/article/Deep_Learning What else have you found particularly useful?
- cs702 10y agowithfries2: Bengio's 2009 survey and Schmidhuber's "conspiracy" blog post contain useful, accurate historical background from the perspective of two leading researchers, with lots of links to additional sources: http://www.iro.umontreal.ca/~bengioy/papers/ftml.pdf http://www.iro.umontreal.ca/~bengioy/papers/ftml.pdf http://people.idsia.ch/~juergen/deep-learning-conspiracy.html http://people.idsia.ch/~juergen/deep-learning-conspiracy.htm... Were I in your shoes, I would NOT have highlighted the Google YouTube experiment as "the" big breakthrough. It was just an interesting worthwhile experiment by one of many groups of talented AI researchers who have made slow progress over decades of hard work. Why single it out? -- PS. The YouTube experiment did not produce new theory, and from a practical standpoint, it would be unfair to say that it reignited interest in deep learning. Consider that the paper currently has only ~800 citations, according to Google Scholar.[1] For comparison, Krizhevsky et al's paper describing the deep net that won Imagenet (trained on one computer with one GPU) has over 5000 citations.[2] And neither of these experiments deserves to be called "the" big breakthrough. [1] https://scholar.google.com/citations?view_op=view_citation&hl=en&user=JgDKULMAAAAJ&citation_for_view=JgDKULMAAAAJ:r0BpntZqJG4C https://scholar.google.com/citations?view_op=view_citation&h... [2] https://scholar.google.com/citations?view_op=view_citation&hl=en&user=xegzhJcAAAAJ&citation_for_view=xegzhJcAAAAJ:u5HHmVD_uO8C https://scholar.google.com/citations?view_op=view_citation&h...
- withfries2 10y agoMostly, I wanted to highlight the importance of scale (data + compute) for the accuracy of deep networks.