5 ms·

It's amazing to step back and look at how much of NVIDIA's success has come from unforeseen directions. For their original purpose of making graphics chips, the

by wtallis 8mo ago

It's amazing to step back and look at how much of NVIDIA's success has come from unforeseen directions. For their original purpose of making graphics chips, the consumer vs pro divide was all about CAD support and optional OpenGL features that games didn't use. Programmable shaders were added for the sake of graphics rendering needs, but ended up spawning the whole GPGPU concept, which NVIDIA reacted to very well with the creation and promotion of CUDA. GPUs have FP64 capabilities in the first place because back when GPGPU first started happening, it was all about traditional HPC workloads like numerical solutions to PDEs. Fast forward several years, and the cryptocurrency craze drove up GPU prices for many years without even touching the floating-point capabilities. Now, FP64 is out because of ML, a field that's almost unrecognizable compared to where it was during the first few years of CUDA's existence. NVIDIA has been very lucky over the course of their history, but have also done a great job of reacting to new workloads and use cases. But those shifts have definitely created some awkward moments where their existing strategies and roadmaps have been upturned.

rustyhancock 8mo ago
They were also bailed out by Sega. When they couldn't deliver the console GPU they promised for the Dreamcast (the NV2), Shoichiro Irimajiri, the Sega CEO at the time let them keep the cash in exchange for stock [0]. Without it Nvidia would have gone bankrupt months before Riva 128 changed things. Sega console arm went bust not that it mattered. But they sold the stock for about $15mn (3x). Had they held it, Jensen Huang ,estimated itd be worth a trillion[1]. Obviously Sega and especially it's console arm wasn't really into VC but... My wet dream has always been what if Sega and Nvidia stuck together and we had a Sega tegra shield instead of a Nintendo switch? Or even what if Sega licensed itself to the Steam Deck? You can tell I'm a sega fan boy but I can't help that the Mega Drive was the first console I owned and loved! [0] https://www.gamespot.com/articles/a-5-million-gift-from-sega-saved-nvidia/1100-6536736/ https://www.gamespot.com/articles/a-5-million-gift-from-sega... [1] https://youtu.be/3hptKYix4X8?t=5483&si=h0sBmIiaduuJiem_ https://youtu.be/3hptKYix4X8?t=5483&si=h0sBmIiaduuJiem_
readitalready 8mo ago
The whole GPU history is off and being driven by finance bros as well. Everyone believes Nvidia kicked off the GPU AI craze when Ilya Sutskever cleaned up on AlexNet with an Nvidia GPU back in 2012, or when Andrew Ng and team at Stanford published their "Large Scale Deep Unsupervised Learning using Graphics Processors" in 2009, but in 2004, a couple of Korean researches were the first to implement neural networks on a GPU, using ATI Radeons (now AMD): https://www.sciencedirect.com/science/article/abs/pii/S0031320304000524 https://www.sciencedirect.com/science/article/abs/pii/S00313... I remember ATI and Nvidia were neck-and-neck to launch the first GPUs around 2000. Just so much happening so fast. I'd also say Nvidia had the benefit of AMD going after and focusing on Intel both at the server level as well as the integrated laptop processors, which was the reason they bought ATI.
gdiamos 8mo ago
Most people don't appreciate how many dead end applications NVIDIA explored before finding deep learning. It took a very long time, and it wasn't luck.
- wtallis 8mo ago
  It was luck that a viable non-graphics application like deep learning existed which was well-suited to the architecture NVIDIA already had on hand. I certainly don't mean to diminish the work NVIDIA did to build their CUDA ecosystem, but without the benefit of hindsight I think it would have been very plausible that GPU architectures would not have been amenable to any use cases that would end up dwarfing graphics itself. There are plenty of architectures in the history of computing which never found a killer application, let alone three or four.
  - oivey 8mo ago
    Even that is arguably not lucky, it just followed a non-obvious trajectory. Graphics uses a fair amount of linear algebra, so people with large scale physical modeling needs (among many) became interested. To an extent the deep learning craze kicked off because of developments in computation on GPUs enabled economical training.
    - imtringued 8mo ago
      Nvidia started their GPGPU adventure by acquiring a physics engine and porting it over to run on their GPUs. Supporting linear algebra operations was pretty much the goal from the start.
      - adrian_b 8mo ago
        They were also full of lies when they have started their GPGPU adventure (like also today). For a few years they have repeated continuously how GPGPU can provide about 100 times more speed than CPUs. This has always been false. GPUs are really much faster, but their performance per watt has oscillated during most of the time around 3 times and sometimes up to 4 times greater in comparison with CPUs. This is impressive, but very far from the "100" factor originally claimed by NVIDIA. Far more annoying than the exaggerated performance claims, is how the NVIDIA CEO was talking during the first GPGPU years about how their GPUs will cause a democratization of computing, giving access for everyone to high-throughput computing. After a few years, these optimistic prophecies have stopped and NVIDIA has promptly removed FP64 support from their price-acceptable GPUs. A few years later, AMD has followed the NVIDIA example. Now, only Intel has made an attempt to revive GPUs as "GPGPUs", but there seems to be little conviction behind this attempt, as they do not even advertise the capabilities of their GPUs. If Intel will also abandon this market, than the "general-purpose" in GPGPUs will really become dead.
pmontra 8mo ago
I don't think it was luck. I think it was inevitable. They positioned the company on high performance computing, even if maybe they didn't think they were a HPC company, and something was bound to happen in that market because everybody was doing more and more computing. Then they executed well with the usual amount of greed that every company has. The only risk for well positioned companies is being too ahead of times: being in the right market but not surviving long enough to see a killer app happen.
brookst 8mo ago
Maybe some luck. But there’s also a principle that if you optimize the hell out of something and follow customer demand, there’s money to be made. Nvidia did a great job of avoiding the “oh we’re not in that market” trap that sunk Intel (phones, GPUs, efficient CPUs). Where Intel was too big and profitable to cultivate adjacent markets, Nvidia did everything they could to serve them and increase demand.
pjc50 8mo ago
The counter question is: why have AMD been so bad by comparison?
- Eisenstein 8mo ago
  Because GPUs require a lot on the software side, and AMD sucks at software. They are a CPU company that bought a GPU company. ATI should have been left alone.
- 0x457 8mo ago
  Because AMD only every had one strategy for software: Look at it as, we're David against the Goliath, we're going to utilize open-source because that's how you win. Then make zero effecting trying to increase adoption, which then result in reducing support for it and eventually giving up.

by wtallis 8mo ago

rustyhancock 8mo ago

They were also bailed out by Sega. When they couldn't deliver the console GPU they promised for the Dreamcast (the NV2), Shoichiro Irimajiri, the Sega CEO at the time let them keep the cash in exchange for stock [0]. Without it Nvidia would have gone bankrupt months before Riva 128 changed things. Sega console arm went bust not that it mattered. But they sold the stock for about $15mn (3x). Had they held it, Jensen Huang ,estimated itd be worth a trillion[1]. Obviously Sega and especially it's console arm wasn't really into VC but... My wet dream has always been what if Sega and Nvidia stuck together and we had a Sega tegra shield instead of a Nintendo switch? Or even what if Sega licensed itself to the Steam Deck? You can tell I'm a sega fan boy but I can't help that the Mega Drive was the first console I owned and loved! [0] https://www.gamespot.com/articles/a-5-million-gift-from-sega-saved-nvidia/1100-6536736/ https://www.gamespot.com/articles/a-5-million-gift-from-sega... [1] https://youtu.be/3hptKYix4X8?t=5483&si=h0sBmIiaduuJiem_ https://youtu.be/3hptKYix4X8?t=5483&si=h0sBmIiaduuJiem_

readitalready 8mo ago

The whole GPU history is off and being driven by finance bros as well. Everyone believes Nvidia kicked off the GPU AI craze when Ilya Sutskever cleaned up on AlexNet with an Nvidia GPU back in 2012, or when Andrew Ng and team at Stanford published their "Large Scale Deep Unsupervised Learning using Graphics Processors" in 2009, but in 2004, a couple of Korean researches were the first to implement neural networks on a GPU, using ATI Radeons (now AMD): https://www.sciencedirect.com/science/article/abs/pii/S0031320304000524 https://www.sciencedirect.com/science/article/abs/pii/S00313... I remember ATI and Nvidia were neck-and-neck to launch the first GPUs around 2000. Just so much happening so fast. I'd also say Nvidia had the benefit of AMD going after and focusing on Intel both at the server level as well as the integrated laptop processors, which was the reason they bought ATI.

gdiamos 8mo ago

Most people don't appreciate how many dead end applications NVIDIA explored before finding deep learning. It took a very long time, and it wasn't luck.

wtallis 8mo ago
It was luck that a viable non-graphics application like deep learning existed which was well-suited to the architecture NVIDIA already had on hand. I certainly don't mean to diminish the work NVIDIA did to build their CUDA ecosystem, but without the benefit of hindsight I think it would have been very plausible that GPU architectures would not have been amenable to any use cases that would end up dwarfing graphics itself. There are plenty of architectures in the history of computing which never found a killer application, let alone three or four.
- oivey 8mo ago
  Even that is arguably not lucky, it just followed a non-obvious trajectory. Graphics uses a fair amount of linear algebra, so people with large scale physical modeling needs (among many) became interested. To an extent the deep learning craze kicked off because of developments in computation on GPUs enabled economical training.
  - imtringued 8mo ago
    Nvidia started their GPGPU adventure by acquiring a physics engine and porting it over to run on their GPUs. Supporting linear algebra operations was pretty much the goal from the start.
    - adrian_b 8mo ago
      They were also full of lies when they have started their GPGPU adventure (like also today). For a few years they have repeated continuously how GPGPU can provide about 100 times more speed than CPUs. This has always been false. GPUs are really much faster, but their performance per watt has oscillated during most of the time around 3 times and sometimes up to 4 times greater in comparison with CPUs. This is impressive, but very far from the "100" factor originally claimed by NVIDIA. Far more annoying than the exaggerated performance claims, is how the NVIDIA CEO was talking during the first GPGPU years about how their GPUs will cause a democratization of computing, giving access for everyone to high-throughput computing. After a few years, these optimistic prophecies have stopped and NVIDIA has promptly removed FP64 support from their price-acceptable GPUs. A few years later, AMD has followed the NVIDIA example. Now, only Intel has made an attempt to revive GPUs as "GPGPUs", but there seems to be little conviction behind this attempt, as they do not even advertise the capabilities of their GPUs. If Intel will also abandon this market, than the "general-purpose" in GPGPUs will really become dead.

pmontra 8mo ago

I don't think it was luck. I think it was inevitable. They positioned the company on high performance computing, even if maybe they didn't think they were a HPC company, and something was bound to happen in that market because everybody was doing more and more computing. Then they executed well with the usual amount of greed that every company has. The only risk for well positioned companies is being too ahead of times: being in the right market but not surviving long enough to see a killer app happen.

brookst 8mo ago

Maybe some luck. But there’s also a principle that if you optimize the hell out of something and follow customer demand, there’s money to be made. Nvidia did a great job of avoiding the “oh we’re not in that market” trap that sunk Intel (phones, GPUs, efficient CPUs). Where Intel was too big and profitable to cultivate adjacent markets, Nvidia did everything they could to serve them and increase demand.

pjc50 8mo ago

The counter question is: why have AMD been so bad by comparison?

Eisenstein 8mo ago
Because GPUs require a lot on the software side, and AMD sucks at software. They are a CPU company that bought a GPU company. ATI should have been left alone.
0x457 8mo ago
Because AMD only every had one strategy for software: Look at it as, we're David against the Goliath, we're going to utilize open-source because that's how you win. Then make zero effecting trying to increase adoption, which then result in reducing support for it and eventually giving up.