4 ms·

A stubborn computer scientist accidentally launched the deep learning boom

76 points by LorenDB 2y ago

deleted 2y ago
[deleted]
qrios 2y ago
> Huang argued that the simple existence of CUDA would enlarge the supercomputing sector. This view was not widely held, and by the end of 2008, Nvidia’s stock price had declined by seventy percent… This is a good example of how investor behaviour can only quantitatively project what will happen in the future. Huang's bet on GPUs for high performance computing made sense in the long term. Intel didn't have the staying power with the i860[1] a decade earlier (and of course had no idea how to offer decent developer tools). I tried really hard to develop meaningful and executable programmes with an 8bit card (DSM). CUDA was a revelation for me. [1] https://www.geekdot.com/intel-80860/ https://www.geekdot.com/intel-80860/
- trynumber9 2y ago
  The stock value didn't drop because of CUDA but because most stocks and tech stocks dropped in 2008. AMD was also down 80%, Intel down 40%.
  - magicalhippo 2y ago
    I was going to say anyone should have noted that stocks being down in 2008 means nothing due to the global financial crisis[1]. But then I realized the author might have been a toddler at that time so might not have lived through it, making it easier to forget. How time flies... [1]: https://en.wikipedia.org/wiki/2007%E2%80%932008_financial_crisis https://en.wikipedia.org/wiki/2007%E2%80%932008_financial_cr...
julianeon 2y ago
She wrote a book incidentally, I haven't read it but I'm sure it's worth checking out: The Worlds I See.
VyseofArcadia 2y ago
> Nvidia invented the GPU in 1999 This is just plain not true. Who edited this? Did no one even bother to skim the Wikipedia page on GPUs?
- parshimers 2y ago
  yeah, you could argue that the RIVA128 was the first successful PC GPU that didn't require a separate video card, but that's not the same thing.
  - p_l 2y ago
    succesful consumer PC GPU, if that.
- baq 2y ago
  > Who edited this? chatgpt, probably.
- f1shy 2y ago
  The whole thing is absolutely utter crap. As early as 2002 there was, in the university where I studied a BIG push into NN. The term “deep learning” is also lousy and marketing.
  - Chabsff 2y ago
    Isn't "deep learning" specifically referring to the breakthroughs that finally started addressing the issues preventing architectures with more than a single hidden layer from achieving anything? Unless I'm getting all of that wrong, it doesn't sound like that bad of a term. I get that it has since been used and abused to absurdity. But it's not like it came out of nowhere at first.
    - mp05 2y ago
      My professor told me learning is considered "deep" if your NN has more than two layers.
    - 2y ago
prettyStandard 2y ago
In college 2004-2008 I was ridiculed a few times for thinking Neural Networks were interesting. The way I saw it was human intelligence was built on neural network. All we were missing were sufficiently accurate models of neurons or scale. In middle school I estimated to rival the processing power of a human brain you would need the processing power of an Empire State building full of Pentium 3 processors. It was a pretty rough calculation. I'm having a hard time remembering just how it went. lol
- neofrommatrix 2y ago
  I had a similar experience when I wanted to focus on NN research in 2007. My advisor had the wrong incentives. FML.
yapyap 2y ago
Lol someone likes Nvidia
davideg 2y ago
The Alignment Problem: Machine Learning and Human Values[1] by Brian Christian is a nice overview of some of this history and exploration of some of the pitfalls in different ML approaches. It's really interesting reading it now given what's been happening with OpenAI and ChatGPT, etc. [1] https://en.wikipedia.org/wiki/The_Alignment_Problem https://en.wikipedia.org/wiki/The_Alignment_Problem
HarHarVeryFunny 2y ago
I may be wrong, but I disagree with a lot of the spin being pushed in this article, although of course Hinton deserves a lot of the credit for keeping the field alive and igniting the "deep learning" explosion. - The ImageNet dataset / competition pre-dates neural net entrants, and I'm not aware that Fei Fei created it in anticipation of such. The reason that AlexNet (2012 ILSVRC/ImageNet entry) made such an impact was that it beat all non-ANN entrants by such a huge margin that it was impossible to ignore, and pretty much killed all ongoing attempts to hand-design transform-invariant image features (such as SIFT). - While NVidia deserve credit for enabling GPU-compute with CUDA, it was pure luck that ANNs subsequently took off, and became the primary use of CUDA (which then added ANN specific libraries such as cuDNN). - AlexNet (ImageNet 2012) was certainly what started the ANN boom, which Hinton/LeCun/Bengio then decided to re-brand as "deep learning" to escape any historically negative association of "neural networks". However, while AlexNet demonstrated the power of a large (& deep) neural net, I don't think it's fair to say that it was responsible the current/waning belief in LLM "scaling laws". The immediate aftermath of AlexNet wasn't bigger datasets, but attempts to build bigger/better ANNs to do better on the ImageNet benchmark. The LLM "scaling laws" originated from OpenAI's GPT-1 and GPT-2 where unexpected model capabilities lead to experimentation of scaling, with Sutskever and Amodei being two of the earliest believers. Sutskever has recently said that he thinks that transformer scaling has plateaued, and has started his own company (SSI) pursuing a different approach. I don't think we can say that Hinton accidentally created the deep learning boom. He always (to his huge credit) believed in ANNs, pushed it into the public eye with AlexNet, created the "Deep Learning" branding, and generally promoted it until it got too powerful for his liking.
- pavon 2y ago
  I don't see any of that in the article, in fact quite the opposite. The article explicitly states that neither Fei Fei nor NVidia were thinking about neural networks when creating ImageNet or CUDA, rather it is spinning the serendipity of the three unrelated efforts coming together to create deep learning, and the idea that none of the three would have happened if it weren't for people pushing ideas that went contrary to conventional wisdom. > I don't think we can say that Hinton accidentally created the deep learning boom. Yeah, the headline is unclear, but I read it as referring to Fei Fei being the "accidental" contributor to deep learning, not Hinton.
trhway 2y ago
in 15 years from : "He then reached out to Nvidia. “I sent an e-mail saying, ‘Look, I just told a thousand machine-learning researchers they should go and buy Nvidia cards. Can you send me a free one?’ ” Hinton told me. “They said no.”" to Altman saying he needs $7T and people eagerly lining up to give the money. Moore law for the venture money in tech. And each time the new tech wave is higher. Somebody somewhere is already working on that next "trillions dollar something in 10-15 years". Quantum computing based NNs? Or may be just robots everywhere.
area51org 2y ago
While I do like a lot of what AI brings us, it also scares the hell out of me and makes me fear more for the future of humanity than I ever have. "Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should." —"Dr. Ian Malcolm" (Jeff Goldblum), Jurassic Park
throw0101b 2y ago
See "Geoffrey Hinton in conversation with Fei-Fei Li" where the two got together and looked back at the developments over the last ~10 years, including why and how they did their part: * https://www.youtube.com/watch?v=QWWgr2rN45o https://www.youtube.com/watch?v=QWWgr2rN45o
BriggyDwiggs42 2y ago
But what im really curious about is whats all them orbs floatin outta them drawers there
- parpfish 2y ago
  That’s what big data storage looked like before we had cloud data warehouses
- tuespetre 2y ago
  Those are for the data wizards to ponder.
btilly 2y ago
I think that this is a general trend. Various fields tend to get into groupthink, and it is hard for people who buck the trend to pursue the usual academic path to research. But usually the herd has gone down a wrong turn, and it is up to a random rebel to make the next key discovery. For an extreme example, I recommend Veratasium's video on blue LEDs, https://www.youtube.com/watch?v=AF8d72mA41M https://www.youtube.com/watch?v=AF8d72mA41M. But despite how extreme that example is, it does seem to be a trend. The direction that the herd has gone has been generally well-explored simply because a herd of researchers have already gone there. If they've found nothing, you'll probably find nothing either. So you have to back up. Sadly, the way that academia is structured discourages this at every turn. :-(
- akamaka 2y ago
  On that note, it’s fun to look at the responses on HN when deep learning made its breakthrough on the ImageNet dataset: https://news.ycombinator.com/item?id=4611830 https://news.ycombinator.com/item?id=4611830
- deanCommie 2y ago
  I think the problem is there is academia the calling, and academia the profession. Academia the calling is the pursuit of knowledge, truth, the advancement of humankind. It sometimes needs flexibility, other times rigor. Sometimes a full team of aligned individuals. Sometimes the singular concentration of one person over a long period of time. Results are not guaranteed, and even when they happen they may be revolutionary, or they may be a tiny stepping stone that only reveals it's greater purpose literally CENTURIES later. (Joseph Fourier died in 1830, never realizing that 2 centuries from his all modern life would not be possible without mathematical concepts named after him) All of this is great, but...how does one survive to be able to do this? Historically, virtually every famous scientist you can think of was a noble. They had family money, servants. They didn't have to think about rent or laundry or cooking or parenting. If instead we want to be able to guide humans with the ABILITY to do great things towards such pursuits, how do we compensate them? How do we track progress, and create reward structures to incentivize both achievement and progress? Therein lies academia the profession. I am not in it, nor have I ever been it. I have nothing to gain from defending it. Every ex-academic in industry I come across is so relieved to be out of it. But that is so sad and disappointing to me, and it pains me how often it's trashed on hackernews...
gregw2 2y ago
This article was discussed last week on HN but based on the author's blog post rather than ArsTechnica who is seemingly syndicating it to a larger audience. Original discussion: https://news.ycombinator.com/item?id=42057139 https://news.ycombinator.com/item?id=42057139
- teleforce 2y ago
  Previous discussions on the same ArsTechnica articles: https://news.ycombinator.com/item?id=42116140 https://news.ycombinator.com/item?id=42116140 https://news.ycombinator.com/item?id=42106623 https://news.ycombinator.com/item?id=42106623 You can check the past discussions of the same article using the past link underneath the title post.
- dang 2y ago
  Thanks! Macroexpanded: The deep learning boom caught almost everyone by surprise - https://news.ycombinator.com/item?id=42057139 https://news.ycombinator.com/item?id=42057139 - Nov 2024 (188 comments)
dantheman252 2y ago
Interesting that this was done in academia. Now days this would just be hidden within some company not useable by others.
- danielmarkbruce 2y ago
  Like transformers. Oh wait.
readyplayernull 2y ago
> Nvidia invented the GPU in 1999
yogrish 2y ago
Now the fourth element is the seminal paper " Attention is all you need" which has taken AI into next level with openAI LLMs and the likes. Another story that rightly fits in here. https://www.ft.com/content/37bb01af-ee46-4483-982f-ef3921436a50 https://www.ft.com/content/37bb01af-ee46-4483-982f-ef3921436...

76 points by LorenDB 2y ago

deleted 2y ago

[deleted]

> Huang argued that the simple existence of CUDA would enlarge the supercomputing sector. This view was not widely held, and by the end of 2008, Nvidia’s stock price had declined by seventy percent… This is a good example of how investor behaviour can only quantitatively project what will happen in the future. Huang's bet on GPUs for high performance computing made sense in the long term. Intel didn't have the staying power with the i860[1] a decade earlier (and of course had no idea how to offer decent developer tools). I tried really hard to develop meaningful and executable programmes with an 8bit card (DSM). CUDA was a revelation for me. [1] https://www.geekdot.com/intel-80860/ https://www.geekdot.com/intel-80860/

trynumber9 2y ago
The stock value didn't drop because of CUDA but because most stocks and tech stocks dropped in 2008. AMD was also down 80%, Intel down 40%.
- magicalhippo 2y ago
  I was going to say anyone should have noted that stocks being down in 2008 means nothing due to the global financial crisis[1]. But then I realized the author might have been a toddler at that time so might not have lived through it, making it easier to forget. How time flies... [1]: https://en.wikipedia.org/wiki/2007%E2%80%932008_financial_crisis https://en.wikipedia.org/wiki/2007%E2%80%932008_financial_cr...

julianeon 2y ago

She wrote a book incidentally, I haven't read it but I'm sure it's worth checking out: The Worlds I See.

VyseofArcadia 2y ago

> Nvidia invented the GPU in 1999 This is just plain not true. Who edited this? Did no one even bother to skim the Wikipedia page on GPUs?

parshimers 2y ago
yeah, you could argue that the RIVA128 was the first successful PC GPU that didn't require a separate video card, but that's not the same thing.
- p_l 2y ago
  succesful consumer PC GPU, if that.
baq 2y ago
> Who edited this? chatgpt, probably.
f1shy 2y ago
The whole thing is absolutely utter crap. As early as 2002 there was, in the university where I studied a BIG push into NN. The term “deep learning” is also lousy and marketing.
- Chabsff 2y ago
  Isn't "deep learning" specifically referring to the breakthroughs that finally started addressing the issues preventing architectures with more than a single hidden layer from achieving anything? Unless I'm getting all of that wrong, it doesn't sound like that bad of a term. I get that it has since been used and abused to absurdity. But it's not like it came out of nowhere at first.
  - mp05 2y ago
    My professor told me learning is considered "deep" if your NN has more than two layers.
  - 2y ago

prettyStandard 2y ago

In college 2004-2008 I was ridiculed a few times for thinking Neural Networks were interesting. The way I saw it was human intelligence was built on neural network. All we were missing were sufficiently accurate models of neurons or scale. In middle school I estimated to rival the processing power of a human brain you would need the processing power of an Empire State building full of Pentium 3 processors. It was a pretty rough calculation. I'm having a hard time remembering just how it went. lol

neofrommatrix 2y ago
I had a similar experience when I wanted to focus on NN research in 2007. My advisor had the wrong incentives. FML.

yapyap 2y ago

Lol someone likes Nvidia

davideg 2y ago

The Alignment Problem: Machine Learning and Human Values[1] by Brian Christian is a nice overview of some of this history and exploration of some of the pitfalls in different ML approaches. It's really interesting reading it now given what's been happening with OpenAI and ChatGPT, etc. [1] https://en.wikipedia.org/wiki/The_Alignment_Problem https://en.wikipedia.org/wiki/The_Alignment_Problem

HarHarVeryFunny 2y ago

I may be wrong, but I disagree with a lot of the spin being pushed in this article, although of course Hinton deserves a lot of the credit for keeping the field alive and igniting the "deep learning" explosion. - The ImageNet dataset / competition pre-dates neural net entrants, and I'm not aware that Fei Fei created it in anticipation of such. The reason that AlexNet (2012 ILSVRC/ImageNet entry) made such an impact was that it beat all non-ANN entrants by such a huge margin that it was impossible to ignore, and pretty much killed all ongoing attempts to hand-design transform-invariant image features (such as SIFT). - While NVidia deserve credit for enabling GPU-compute with CUDA, it was pure luck that ANNs subsequently took off, and became the primary use of CUDA (which then added ANN specific libraries such as cuDNN). - AlexNet (ImageNet 2012) was certainly what started the ANN boom, which Hinton/LeCun/Bengio then decided to re-brand as "deep learning" to escape any historically negative association of "neural networks". However, while AlexNet demonstrated the power of a large (& deep) neural net, I don't think it's fair to say that it was responsible the current/waning belief in LLM "scaling laws". The immediate aftermath of AlexNet wasn't bigger datasets, but attempts to build bigger/better ANNs to do better on the ImageNet benchmark. The LLM "scaling laws" originated from OpenAI's GPT-1 and GPT-2 where unexpected model capabilities lead to experimentation of scaling, with Sutskever and Amodei being two of the earliest believers. Sutskever has recently said that he thinks that transformer scaling has plateaued, and has started his own company (SSI) pursuing a different approach. I don't think we can say that Hinton accidentally created the deep learning boom. He always (to his huge credit) believed in ANNs, pushed it into the public eye with AlexNet, created the "Deep Learning" branding, and generally promoted it until it got too powerful for his liking.

pavon 2y ago
I don't see any of that in the article, in fact quite the opposite. The article explicitly states that neither Fei Fei nor NVidia were thinking about neural networks when creating ImageNet or CUDA, rather it is spinning the serendipity of the three unrelated efforts coming together to create deep learning, and the idea that none of the three would have happened if it weren't for people pushing ideas that went contrary to conventional wisdom. > I don't think we can say that Hinton accidentally created the deep learning boom. Yeah, the headline is unclear, but I read it as referring to Fei Fei being the "accidental" contributor to deep learning, not Hinton.

trhway 2y ago

in 15 years from : "He then reached out to Nvidia. “I sent an e-mail saying, ‘Look, I just told a thousand machine-learning researchers they should go and buy Nvidia cards. Can you send me a free one?’ ” Hinton told me. “They said no.”" to Altman saying he needs $7T and people eagerly lining up to give the money. Moore law for the venture money in tech. And each time the new tech wave is higher. Somebody somewhere is already working on that next "trillions dollar something in 10-15 years". Quantum computing based NNs? Or may be just robots everywhere.

area51org 2y ago

While I do like a lot of what AI brings us, it also scares the hell out of me and makes me fear more for the future of humanity than I ever have. "Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should." —"Dr. Ian Malcolm" (Jeff Goldblum), Jurassic Park

throw0101b 2y ago

See "Geoffrey Hinton in conversation with Fei-Fei Li" where the two got together and looked back at the developments over the last ~10 years, including why and how they did their part: * https://www.youtube.com/watch?v=QWWgr2rN45o https://www.youtube.com/watch?v=QWWgr2rN45o

BriggyDwiggs42 2y ago

But what im really curious about is whats all them orbs floatin outta them drawers there

parpfish 2y ago
That’s what big data storage looked like before we had cloud data warehouses
tuespetre 2y ago
Those are for the data wizards to ponder.

btilly 2y ago

I think that this is a general trend. Various fields tend to get into groupthink, and it is hard for people who buck the trend to pursue the usual academic path to research. But usually the herd has gone down a wrong turn, and it is up to a random rebel to make the next key discovery. For an extreme example, I recommend Veratasium's video on blue LEDs, https://www.youtube.com/watch?v=AF8d72mA41M https://www.youtube.com/watch?v=AF8d72mA41M. But despite how extreme that example is, it does seem to be a trend. The direction that the herd has gone has been generally well-explored simply because a herd of researchers have already gone there. If they've found nothing, you'll probably find nothing either. So you have to back up. Sadly, the way that academia is structured discourages this at every turn. :-(

akamaka 2y ago
On that note, it’s fun to look at the responses on HN when deep learning made its breakthrough on the ImageNet dataset: https://news.ycombinator.com/item?id=4611830 https://news.ycombinator.com/item?id=4611830
deanCommie 2y ago
I think the problem is there is academia the calling, and academia the profession. Academia the calling is the pursuit of knowledge, truth, the advancement of humankind. It sometimes needs flexibility, other times rigor. Sometimes a full team of aligned individuals. Sometimes the singular concentration of one person over a long period of time. Results are not guaranteed, and even when they happen they may be revolutionary, or they may be a tiny stepping stone that only reveals it's greater purpose literally CENTURIES later. (Joseph Fourier died in 1830, never realizing that 2 centuries from his all modern life would not be possible without mathematical concepts named after him) All of this is great, but...how does one survive to be able to do this? Historically, virtually every famous scientist you can think of was a noble. They had family money, servants. They didn't have to think about rent or laundry or cooking or parenting. If instead we want to be able to guide humans with the ABILITY to do great things towards such pursuits, how do we compensate them? How do we track progress, and create reward structures to incentivize both achievement and progress? Therein lies academia the profession. I am not in it, nor have I ever been it. I have nothing to gain from defending it. Every ex-academic in industry I come across is so relieved to be out of it. But that is so sad and disappointing to me, and it pains me how often it's trashed on hackernews...

gregw2 2y ago

This article was discussed last week on HN but based on the author's blog post rather than ArsTechnica who is seemingly syndicating it to a larger audience. Original discussion: https://news.ycombinator.com/item?id=42057139 https://news.ycombinator.com/item?id=42057139

teleforce 2y ago
Previous discussions on the same ArsTechnica articles: https://news.ycombinator.com/item?id=42116140 https://news.ycombinator.com/item?id=42116140 https://news.ycombinator.com/item?id=42106623 https://news.ycombinator.com/item?id=42106623 You can check the past discussions of the same article using the past link underneath the title post.
dang 2y ago
Thanks! Macroexpanded: The deep learning boom caught almost everyone by surprise - https://news.ycombinator.com/item?id=42057139 https://news.ycombinator.com/item?id=42057139 - Nov 2024 (188 comments)

dantheman252 2y ago

Interesting that this was done in academia. Now days this would just be hidden within some company not useable by others.

danielmarkbruce 2y ago
Like transformers. Oh wait.

readyplayernull 2y ago

> Nvidia invented the GPU in 1999

yogrish 2y ago

Now the fourth element is the seminal paper " Attention is all you need" which has taken AI into next level with openAI LLMs and the likes. Another story that rightly fits in here. https://www.ft.com/content/37bb01af-ee46-4483-982f-ef3921436a50 https://www.ft.com/content/37bb01af-ee46-4483-982f-ef3921436...