28 ms·

> NVIDIA’s current growth spurt is based on big bet that Huang made a few years ago, when he understood that he was in a unique position to help advance AI, mac

> NVIDIA’s current growth spurt is based on big bet that Huang made a few years ago, when he understood that he was in a unique position to help advance AI, machine learning and deep learning. What kind of bullshit is this. Alex Krizhevsky used CUDA to run neural nets much faster than you could on CPU, blew up ImageNet and NVIDIA found out they were sitting on a goldmine powered by a revolution they really had nothing to do with. NVIDIA really had nothing to do with any of this taking off, except happening to provide the hardware.

llukas 9y ago
Sir, you contradict yourself - CUDA is software and NVIDIA invested in CUDA long time before anybody believed it would take off.
- arcanus 9y ago
  Cuda is a GPGPU paradigm. It was certainly not an investment in machine learning.
  - dragandj 9y ago
    But all major Deep Learning frameworks use Nvidia-s cuDNN library. They provide lots of sugar, but the central thing is developed (or paid for if you prefer) by Nvidia.
  - nightski 9y ago
    Seeing as this https://developer.nvidia.com/cudnn https://developer.nvidia.com/cudnn exists and is widely used I'd say that isn't really true.
- frozenport 9y ago
  A lot of people thought it woukd take off! Hell people were doing GPGPU with ps2!
- DannyBee 9y ago
  Uh, no. Lots of people invested in things like cuda at roughly the same time. CUDA was also very clearly built to enable general fast math computation, and AI came much much much later As GP says, this is some interesting revisionist history. NVidia figured this out at the same time as everyone else.
  - llukas 9y ago
    I still can run early CUDA programs unmodified. This is investment. edit: on newest hardware, without recompilation
- deleted 9y ago
  [deleted]
isatty 9y ago
Pretty much everything this comment said. Nvidia had nothing to do with the first few uses of CUDA for machine learning. It could be done on FPGAs and stuff but that wasn't easily accessible to the masses.
- gok 9y ago
  It was accessible because they invested in CUDA. That wasn’t a small decision.
gok 9y ago
Providing a really well supported platform (CUDA) which only had payoff far in the future did take some foresight. That it could be applied to ML was a good guess, but the fact that the DL revolution happened mostly on NV hardware wasn’t just because the hardware was good. The CUDA environment investment paid off.
ms013 9y ago
Exactly! Plus, even before CUDA existed, people in numerical computing had already noticed that GPUs were ideal hardware for certain classes of problems and were doing awkward things like embedding numerical calculations as shaders and other tricks to use the graphics APIs as a way to encode computations. I recall colleagues who worked on those pre-CUDA systems meeting some resistance from NVIDIA in terms of opening up the low-level APIs that would make the hardware more directly accessible, versus the encodings that they needed to go through to use graphics primitives for the work. NVIDIA got lucky by having the right kind of hardware, and created CUDA not due to some insight into the AI world, but based on the needs of scientific computing. Just look at their early papers on CUDA (E.g., Luebke's 2008 paper "CUDA: SCALABLE PARALLEL PROGRAMMING FOR HIGH-PERFORMANCE SCIENTIFIC COMPUTING") - AI was nowhere to be seen. They were focused on traditional high performance computing problems.
- dragandj 9y ago
  HPC techniques are exactly those that ML and deep learning use in implementation. I don't see how DL is much different from this point of view. It relies on fast linear algebra. HPC relies on fast linear algebra. Nvidia perfected the processor that does this job orders of magnitude faster and provided convenient libraries and development environment to make it accessible.
  - ms013 9y ago
    Nobody is arguing that not to be true. The issue is arguing that NVIDIA foresaw their systems being relevant to AI/DL/ML: they simply were lucky. They opened their architecture to HPC people, concerned largely with fluid dynamics, finite element methods, nbody methods, and so on. To claim that they had any insight into the emergence of ML as a core application of their silicon is simply a laughable revision of history: as we were talking about in an ancestor of this thread.
    - dragandj 9y ago
      Well, I argue that they did foresaw its business viability. They invested heavily in DL before any one of their competitors. Of course there were researchers everywhere working on that, but there are researchers working on everything. What is the most difficult part is recognizing that it can be taken to mainstream and betting huge resources on that. Of course no one argues that Nvidia CEO is the first man in the world that saw GPU/AI applicability. But of those in positions of power in the tech business community...
kayoone 9y ago
But that does not really negate the quote does it? Based on new research in AI, he realized that he was in a unique position and bet a lot of R&D on products that came out years later. Of course he was lucky, but he could have mostly ignored ML and have lost a huge opportunity.
- dragandj 9y ago
  Exactly that! Why the CEO of Intel or the CEO of AMD didn't do the same? It was obvious! Well, it is obvious now, but then it was far from obvious.
Teodolfo 9y ago
In 2009 NVIDIA wouldn't give Geoff Hinton's group the time of day, let alone a single free GPU to try out to see if they wanted to buy more. Geoff Hinton spoke at a NIPS keynote in 2009 and told 1000 people in the audience to all go out and buy a specific NVIDIA card if they wanted to do work in this area and still NVIDIA didn't bat an eye.
godelmachine 9y ago
That's what I thought as soon as I read the opening sentence. NVIDIA founders, at best, through of making dedicated processors for high quality graphics. They certainly weren't aware of AI back then.

llukas 9y ago

Sir, you contradict yourself - CUDA is software and NVIDIA invested in CUDA long time before anybody believed it would take off.

arcanus 9y ago
Cuda is a GPGPU paradigm. It was certainly not an investment in machine learning.
- dragandj 9y ago
  But all major Deep Learning frameworks use Nvidia-s cuDNN library. They provide lots of sugar, but the central thing is developed (or paid for if you prefer) by Nvidia.
- nightski 9y ago
  Seeing as this https://developer.nvidia.com/cudnn https://developer.nvidia.com/cudnn exists and is widely used I'd say that isn't really true.
frozenport 9y ago
A lot of people thought it woukd take off! Hell people were doing GPGPU with ps2!
DannyBee 9y ago
Uh, no. Lots of people invested in things like cuda at roughly the same time. CUDA was also very clearly built to enable general fast math computation, and AI came much much much later As GP says, this is some interesting revisionist history. NVidia figured this out at the same time as everyone else.
- llukas 9y ago
  I still can run early CUDA programs unmodified. This is investment. edit: on newest hardware, without recompilation
deleted 9y ago
[deleted]

isatty 9y ago

Pretty much everything this comment said. Nvidia had nothing to do with the first few uses of CUDA for machine learning. It could be done on FPGAs and stuff but that wasn't easily accessible to the masses.

gok 9y ago
It was accessible because they invested in CUDA. That wasn’t a small decision.

gok 9y ago

Providing a really well supported platform (CUDA) which only had payoff far in the future did take some foresight. That it could be applied to ML was a good guess, but the fact that the DL revolution happened mostly on NV hardware wasn’t just because the hardware was good. The CUDA environment investment paid off.

ms013 9y ago

Exactly! Plus, even before CUDA existed, people in numerical computing had already noticed that GPUs were ideal hardware for certain classes of problems and were doing awkward things like embedding numerical calculations as shaders and other tricks to use the graphics APIs as a way to encode computations. I recall colleagues who worked on those pre-CUDA systems meeting some resistance from NVIDIA in terms of opening up the low-level APIs that would make the hardware more directly accessible, versus the encodings that they needed to go through to use graphics primitives for the work. NVIDIA got lucky by having the right kind of hardware, and created CUDA not due to some insight into the AI world, but based on the needs of scientific computing. Just look at their early papers on CUDA (E.g., Luebke's 2008 paper "CUDA: SCALABLE PARALLEL PROGRAMMING FOR HIGH-PERFORMANCE SCIENTIFIC COMPUTING") - AI was nowhere to be seen. They were focused on traditional high performance computing problems.

dragandj 9y ago
HPC techniques are exactly those that ML and deep learning use in implementation. I don't see how DL is much different from this point of view. It relies on fast linear algebra. HPC relies on fast linear algebra. Nvidia perfected the processor that does this job orders of magnitude faster and provided convenient libraries and development environment to make it accessible.
- ms013 9y ago
  Nobody is arguing that not to be true. The issue is arguing that NVIDIA foresaw their systems being relevant to AI/DL/ML: they simply were lucky. They opened their architecture to HPC people, concerned largely with fluid dynamics, finite element methods, nbody methods, and so on. To claim that they had any insight into the emergence of ML as a core application of their silicon is simply a laughable revision of history: as we were talking about in an ancestor of this thread.
  - dragandj 9y ago
    Well, I argue that they did foresaw its business viability. They invested heavily in DL before any one of their competitors. Of course there were researchers everywhere working on that, but there are researchers working on everything. What is the most difficult part is recognizing that it can be taken to mainstream and betting huge resources on that. Of course no one argues that Nvidia CEO is the first man in the world that saw GPU/AI applicability. But of those in positions of power in the tech business community...

kayoone 9y ago

But that does not really negate the quote does it? Based on new research in AI, he realized that he was in a unique position and bet a lot of R&D on products that came out years later. Of course he was lucky, but he could have mostly ignored ML and have lost a huge opportunity.

dragandj 9y ago
Exactly that! Why the CEO of Intel or the CEO of AMD didn't do the same? It was obvious! Well, it is obvious now, but then it was far from obvious.

Teodolfo 9y ago

In 2009 NVIDIA wouldn't give Geoff Hinton's group the time of day, let alone a single free GPU to try out to see if they wanted to buy more. Geoff Hinton spoke at a NIPS keynote in 2009 and told 1000 people in the audience to all go out and buy a specific NVIDIA card if they wanted to do work in this area and still NVIDIA didn't bat an eye.

godelmachine 9y ago

That's what I thought as soon as I read the opening sentence. NVIDIA founders, at best, through of making dedicated processors for high quality graphics. They certainly weren't aware of AI back then.