15 ms·

I may be wrong, but I disagree with a lot of the spin being pushed in this article, although of course Hinton deserves a lot of the credit for keeping the field

I may be wrong, but I disagree with a lot of the spin being pushed in this article, although of course Hinton deserves a lot of the credit for keeping the field alive and igniting the "deep learning" explosion. - The ImageNet dataset / competition pre-dates neural net entrants, and I'm not aware that Fei Fei created it in anticipation of such. The reason that AlexNet (2012 ILSVRC/ImageNet entry) made such an impact was that it beat all non-ANN entrants by such a huge margin that it was impossible to ignore, and pretty much killed all ongoing attempts to hand-design transform-invariant image features (such as SIFT). - While NVidia deserve credit for enabling GPU-compute with CUDA, it was pure luck that ANNs subsequently took off, and became the primary use of CUDA (which then added ANN specific libraries such as cuDNN). - AlexNet (ImageNet 2012) was certainly what started the ANN boom, which Hinton/LeCun/Bengio then decided to re-brand as "deep learning" to escape any historically negative association of "neural networks". However, while AlexNet demonstrated the power of a large (& deep) neural net, I don't think it's fair to say that it was responsible the current/waning belief in LLM "scaling laws". The immediate aftermath of AlexNet wasn't bigger datasets, but attempts to build bigger/better ANNs to do better on the ImageNet benchmark. The LLM "scaling laws" originated from OpenAI's GPT-1 and GPT-2 where unexpected model capabilities lead to experimentation of scaling, with Sutskever and Amodei being two of the earliest believers. Sutskever has recently said that he thinks that transformer scaling has plateaued, and has started his own company (SSI) pursuing a different approach. I don't think we can say that Hinton accidentally created the deep learning boom. He always (to his huge credit) believed in ANNs, pushed it into the public eye with AlexNet, created the "Deep Learning" branding, and generally promoted it until it got too powerful for his liking.

pavon 2y ago
I don't see any of that in the article, in fact quite the opposite. The article explicitly states that neither Fei Fei nor NVidia were thinking about neural networks when creating ImageNet or CUDA, rather it is spinning the serendipity of the three unrelated efforts coming together to create deep learning, and the idea that none of the three would have happened if it weren't for people pushing ideas that went contrary to conventional wisdom. > I don't think we can say that Hinton accidentally created the deep learning boom. Yeah, the headline is unclear, but I read it as referring to Fei Fei being the "accidental" contributor to deep learning, not Hinton.

pavon 2y ago

I don't see any of that in the article, in fact quite the opposite. The article explicitly states that neither Fei Fei nor NVidia were thinking about neural networks when creating ImageNet or CUDA, rather it is spinning the serendipity of the three unrelated efforts coming together to create deep learning, and the idea that none of the three would have happened if it weren't for people pushing ideas that went contrary to conventional wisdom. > I don't think we can say that Hinton accidentally created the deep learning boom. Yeah, the headline is unclear, but I read it as referring to Fei Fei being the "accidental" contributor to deep learning, not Hinton.