5 ms·

Yep, I checked out at: "...rise of deep learning since 2013, more or less when Google’s X Lab developed a machine learning algorithm able to autonomously browse

Yep, I checked out at: "...rise of deep learning since 2013, more or less when Google’s X Lab developed a machine learning algorithm able to autonomously browse YouTube to identify the videos that contained cats". First, this started in 2012. Second, it wasn't Google - it was when Krizhevsky et al published their seminal work. Realistically, Google was slow to adopt to GPUs at the time, which I understand even contributed to Prof Ng's departure. It was Baidu who launched the first large scale deep-learning based image search, well ahead of Google. Google has certainly caught up, but nobody can say they started it (and be taken seriously).

daveguy 10y ago
Technically deep learning started before 2008. Here is a trends paper from back then: http://www.cs.toronto.edu/~fritz/absps/tics.pdf http://www.cs.toronto.edu/~fritz/absps/tics.pdf Here is a google tech talk from 2007: https://www.youtube.com/watch?v=AyzOUbkUf3M https://www.youtube.com/watch?v=AyzOUbkUf3M Companies didn't pick it up until more recently. GPU-ification happened in 2009 with Ng's group: http://robotics.stanford.edu/~ang/papers/icml09-LargeScaleUnsupervisedDeepLearningGPU.pdf http://robotics.stanford.edu/~ang/papers/icml09-LargeScaleUn... And yes, Krizhevsky et al (Hinton's lab) applied GPU deep learning to ImageNet in 2010: https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf https://papers.nips.cc/paper/4824-imagenet-classification-wi...
- strebler 10y ago
  Those are great links! But the 2008 Hinton paper would not be considered deep learning, it is classic neural nets. It makes no mention of CNNs or GPUs, which is what really got this all going back in 2012 with ImageNet / Krizhevsky. The ImageNet paper is from 2012, not 2010. That's when the computer vision community really went "wow". IIRC, almost every entry in ImageNet 2013 was using CNNs.
  - mattkrause 10y ago
    > it is classic neural nets. It makes no mention of CNNs or GPUs Is using a GPU "essential" for something to be deep learning? I'd always thought that the important part was some sort of hierarchical representation learning. GPUs certainly help, in that you don't want to wait all day while your code does that, but they're not necessary.
    - edfernandez 10y ago
      I think Tsvi Achler's video here will be useful to understand better what the article is about https://www.youtube.com/watch?v=9gTJorBeLi8 https://www.youtube.com/watch?v=9gTJorBeLi8
  - daveguy 10y ago
    Good call on the 2012 not 2010 date. I missed that. GPU are not requirements of deep nn. Hinton's pseudo-bayesian + ReLU approach was the last piece of the deep neural net functionality. CNNs dated back to 1995-1998 with LeCun and Bengio. Although GPUs do accelerate deep NNs enough to be feasible on image data (thanks to Ng).

daveguy 10y ago

Technically deep learning started before 2008. Here is a trends paper from back then: http://www.cs.toronto.edu/~fritz/absps/tics.pdf http://www.cs.toronto.edu/~fritz/absps/tics.pdf Here is a google tech talk from 2007: https://www.youtube.com/watch?v=AyzOUbkUf3M https://www.youtube.com/watch?v=AyzOUbkUf3M Companies didn't pick it up until more recently. GPU-ification happened in 2009 with Ng's group: http://robotics.stanford.edu/~ang/papers/icml09-LargeScaleUnsupervisedDeepLearningGPU.pdf http://robotics.stanford.edu/~ang/papers/icml09-LargeScaleUn... And yes, Krizhevsky et al (Hinton's lab) applied GPU deep learning to ImageNet in 2010: https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf https://papers.nips.cc/paper/4824-imagenet-classification-wi...

strebler 10y ago
Those are great links! But the 2008 Hinton paper would not be considered deep learning, it is classic neural nets. It makes no mention of CNNs or GPUs, which is what really got this all going back in 2012 with ImageNet / Krizhevsky. The ImageNet paper is from 2012, not 2010. That's when the computer vision community really went "wow". IIRC, almost every entry in ImageNet 2013 was using CNNs.
- mattkrause 10y ago
  > it is classic neural nets. It makes no mention of CNNs or GPUs Is using a GPU "essential" for something to be deep learning? I'd always thought that the important part was some sort of hierarchical representation learning. GPUs certainly help, in that you don't want to wait all day while your code does that, but they're not necessary.
  - edfernandez 10y ago
    I think Tsvi Achler's video here will be useful to understand better what the article is about https://www.youtube.com/watch?v=9gTJorBeLi8 https://www.youtube.com/watch?v=9gTJorBeLi8
- daveguy 10y ago
  Good call on the 2012 not 2010 date. I missed that. GPU are not requirements of deep nn. Hinton's pseudo-bayesian + ReLU approach was the last piece of the deep neural net functionality. CNNs dated back to 1995-1998 with LeCun and Bengio. Although GPUs do accelerate deep NNs enough to be feasible on image data (thanks to Ng).