6 ms·

Are you saying that Google, Meta, Amazon, etc... can't do software? It's the bread and butter of these companies. The CUDA moat is important to hold off the lik

by matwood 11mo ago

Are you saying that Google, Meta, Amazon, etc... can't do software? It's the bread and butter of these companies. The CUDA moat is important to hold off the likes of AMD, but hardware like TPUs for internal use or other big software makers is not a big hurdle. Of course Huang will lean on the software being key because he sees the hardware competition catching up.

sigmoid10 11mo ago
Huang said that many years ago, long before ChatGPT or the current AI hype were a thing. In that interview he said that their costs for software R&D and support are equal or even bigger than their hardware side. They've also been hiring top SWE talent for almost two decades now. None of the other companies have spent even close to this much time and money on GPU software, at least until LLMs became insanely popular. So I'd be surprised to see them catch up anytime soon.
qdotme 11mo ago
Essentially, yes, they haven’t done deep software. Netflix probably comes closest amongst FAANG. Google, Meta, Amazon do “shallow and broad” software. They are quite fast at capturing new markets swiftly, they frequently repackage OpenSource core and add the large amount of business logic to make it work, but essentially follow the market cycles - they hire and layoff on a few year cycle, and the people who work there typically also will jump around industries due to both transferable skills and relatively competitive competitors. NVDA is roughly in the same bucket as HFT vendors. They retain talent on a 5-10y timescales. They build software stacks that range from complex kernel drivers and hardware simulators all the way to optimizing compilers and acceleration libraries. This means they can build more integrated, more optimal and more coherent solutions. Just like Tesla can build a more integrated vehicle than Ford.
- thaumasiotes 11mo ago
  But the first example sigmoid10 gave of a company that can't do software was Microsoft.
  - OccamsMirror 11mo ago
    Yeah I'm not convinced Microsoft can do software anymore. I think they're a shambling mess of a zombie software company with enough market entropy to keep going for a long time.
    - Eggpants 11mo ago
      Yeah the fact they had to resort to forking Chrome because they couldn’t engineer a browser folks wanted to use is pretty telling.
      - hollerith 11mo ago
        I don't think that tells us anything. Maintaining a web browser requires about 1000 full-time developers (about the size of the Chrome team at Google) i.e., about $400 million a year. Why would Microsoft incur that cost when Chromium is available under a license that allows Microsoft to do whatever it wants with it?
        Eggpants 11mo ago
        You could say the same thing about all Microsoft products then. How many full time developers does it take to support Windows 11 when Linux is available, SqlServer when Postgres is available, Office when LibreOffice exists? And so on all under licenses that allows Microsoft do whatever it wants with? They should be embarrassed to do better, not spin it into a “wise business move” aka transfer that money into executive bonuses.
        hollerith 11mo ago
        Microsoft gets a lot of its revenue from the sale of licenses and subscriptions for Windows and Office. An unreliable source that gives fast answers to questions tells me that the segments responsible for those two softwares have revenue of about $13 and about 20 billion per quarter respectively. In contrast, basically no one derives any significant revenue from the sale of licenses or subscriptions for web browsers. As long as Microsoft can modify Chromium to have Microsoft's branding, to nag the user into using Microsoft Copilot and to direct search queries to Bing instead of Google Search, why should Microsoft care about web browsers? It gets worse. Any browser Microsoft offers needs to work well on almost any web site. These web sites (of which there are 100s of 1000s) in turn are maintained by developers (hi, web devs!) that tend to be eager to embrace any new technology Google puts into Chrome, with the result that Microsoft must responding by putting the same technological capabilities into its own web browser. Note that the same does not hold for Windows: there is no competitor to Microsoft offering a competitor to Windows that is constantly inducing the maintainers of Windows applications to embrace new technologies, requiring Microsoft to incur the expense of applying engineering pressure to Windows to keep up. This suggests to me that maintaining Windows is actually significantly cheaper than it would be to maintain an independent mainstream browser. An independent mainstream browser is probably the most expensive category of software to create and to maintain excepting only foundational AI models. "Independent" here means "not a fork of Chromium or Firefox". "Mainstream" means "capable of correctly rendering the vast majority of web sites a typical person might want to visit".
- danielscrubs 11mo ago
  Well put. I haven’t thought about it like that.
- musebox35 11mo ago
  I have deep respect for cuda and Nvidia engineering. However, the arguments above seem to totally ignore Google Search indexing and query software stack. They are the king of distributed software and also hardware that scales. That is way TPUs are a thing now and they can compute with Nvidia where AMD failed. Distributed software is the bread and butter of Google with their multi-decade investment from day zero out of necessity. When you have to update an index of an evolving set of billions of documents daily and do that online while keeping subsecond query capability across the globe, that should teach you a few things about deep software stacks.
- bluelightning2k 11mo ago
  You're suggesting Waymo isn't deep software? Or Tensorflow? Or Android? The Go programming language? Or MapReduce, AlphaGo, Kubernetes, the transformer, Chrome/Chromium or Gvisor? You must have an amazing CV to think these are shallow projects.
  - qdotme 11mo ago
    No, I just realize these for what they are - reasonable projects at the exploitation (rather than exploration) stage of any industry. I’d say I have an average CV in the EECS world, but also relatively humble perspective of what is and isn’t bleeding edge. And as the industry expands, the volume „inside” the bleeding edge is exploitation, while the surface is the exploration. Waymo? Maybe; but that’s acquisition and they haven’t done much deep work since. Tensorflow is a handy and very useful DSL, but one that is shallow (builds heavily on CUDA and TPUs etc); Android is another acquisition, and rather incremental growth since; Go is a nth C-like language (so neither Dennis Richie nor Bjarne Stroustrup level work); MapReduce is a darn common concept in HPC (SGI had libraries for it in the 1990s) and implementation was pretty average. AlphaGo - another acquisition, and not much deep work since; Kubernetes is a layer over Linux Namespaces to solve - well - shallow and broad problems; Chrome/Chromium is the 4th major browser that reached dominance and essentially anyone with a 1B to spare can build one.. gVisor is another thin, shallow layer. What I mean by deep software, is a product that requires 5-10y of work before it is useful, that touches multiple layers of software stack (ideally all from hardware to application) etc. But these types of jobs are relatively rare in the 2020s software world (pretty common in robotics and new space) - they were common in the 1990s where I got my calibration values ;) Netscape and Palm Pilot was a „whoa”. Chromium and Android are evolutions.
    - SR2Z 11mo ago
      > No, I just realize these for what they are - reasonable projects at the exploitation (rather than exploration) stage of any industry. I get that bashing on Google is fun, but TensorFlow was the FIRST modern end-user ML library. JAX, an optimizing backend for it, is in its own league even today. The damn thing is almost ten years old already! Waymo is literally the only truly publicly available robotaxi company. I don't know where you get the idea that it's an acquisition; it's the spun-off incarnation of the Google self-driving car project that for years was the butt of "haha, software engineers think they're real engineers" jokes. Again, more than a decade of development on this. Kubernetes is a refinement of Borg, which Google was using to do containerized workloads all the way back in 2003! How's that not a deep project?
- dumah 11mo ago
  These companies innovate in all of those areas and direct those resources towards building hyper-scale custom infrastructure, including CPU, TPU, GPU, and custom networking hardware for the largest cloud systems, and conduct research and development on new compilers and operating system components to exploit them. They're building it for themselves and employ world-class experts across the entire stack. How can NVIDIA develop "more integrated" solutions when they are primarily building for these companies, as well as many others? Examples of these companies doing things you mention as being somehow unique to or characteristic of NVIDIA: Complex kernel drivers or modules: - AWS: Nitro, ENA/EFA, Firecracker, NKI, bottlerocket - Google: gasket/apex, gve, binder - Meta: Katran, bpfilter, cgroup2, oomd, btrfs Hardware simulators: - AWS: Neuron, Annapurna builds simulations for nitro, graviton, inferentia and validates aws instances built for EDA services - Google: Goldfish, Ranchu, Cuttlefish - Meta: Arcadia, MTIA, CFD for thermal management Optimizing Compilers: - Amazon: NNVM, Neo-AI - Google: MLIR, XLA, IREE - Meta: Glow, Triton, LLM Compiler Acceleration Libraries: - Amazon: NeuronX, aws-ofi-nccl - Google: Jax, TF - Meta: FBGEMM, QNNPACK
whywhywhywhy 11mo ago
If CUDA were as trivial to replicate as you say then Nvidia wouldn’t be what it is today.
- whatevaa 11mo ago
  CUDA is not hard to replicate, but the network effects make it very hard to break trough with new product. Just like with everything when network effeft applies.
Miraste 11mo ago
Meta makes websites and apps. Historically, they haven't succeeded at lower-level development. A somewhat recent example was when they tried to make a custom OS for their VR headsets, completely failed, and had to continue using Android.
- coredog64 11mo ago
  Remind me which company originated PyTorch?
  - kanbankaren 11mo ago
    Remind me that PyTorch is not a GPU driver.
- dumah 11mo ago
  You're generalizing a failure at delivering one consumer solution and ignoring the successful infrastructure research and development that occurs behind the scenes. Meta builds hardware from chip to cluster to datacenter scale, and drives research into simulation at every scale, all the way to CFD simulation of datacenter thermal management.
  - Miraste 11mo ago
    More than one failure. They had a project to make a custom chip for model training a few years ago, and they scrapped it. Now they have another one, which entered testing in March. I don't think it's going well, because testing should have wrapped up recently, right before the news that they're in serious talks to buy a lot of TPUs from Google. On the other side of the stack, Llama 4 was a disaster and they haven't shipped anything since. They have the money and talent to do it. As you point out, they do have major successes in areas that take real engineering. But they also have a lot of failures. It will depend how the internal politics play out, I imagine.

by matwood 11mo ago

sigmoid10 11mo ago

Huang said that many years ago, long before ChatGPT or the current AI hype were a thing. In that interview he said that their costs for software R&D and support are equal or even bigger than their hardware side. They've also been hiring top SWE talent for almost two decades now. None of the other companies have spent even close to this much time and money on GPU software, at least until LLMs became insanely popular. So I'd be surprised to see them catch up anytime soon.

qdotme 11mo ago

Essentially, yes, they haven’t done deep software. Netflix probably comes closest amongst FAANG. Google, Meta, Amazon do “shallow and broad” software. They are quite fast at capturing new markets swiftly, they frequently repackage OpenSource core and add the large amount of business logic to make it work, but essentially follow the market cycles - they hire and layoff on a few year cycle, and the people who work there typically also will jump around industries due to both transferable skills and relatively competitive competitors. NVDA is roughly in the same bucket as HFT vendors. They retain talent on a 5-10y timescales. They build software stacks that range from complex kernel drivers and hardware simulators all the way to optimizing compilers and acceleration libraries. This means they can build more integrated, more optimal and more coherent solutions. Just like Tesla can build a more integrated vehicle than Ford.

thaumasiotes 11mo ago
But the first example sigmoid10 gave of a company that can't do software was Microsoft.
- OccamsMirror 11mo ago
  Yeah I'm not convinced Microsoft can do software anymore. I think they're a shambling mess of a zombie software company with enough market entropy to keep going for a long time.
  - Eggpants 11mo ago
    Yeah the fact they had to resort to forking Chrome because they couldn’t engineer a browser folks wanted to use is pretty telling.
    - hollerith 11mo ago
      I don't think that tells us anything. Maintaining a web browser requires about 1000 full-time developers (about the size of the Chrome team at Google) i.e., about $400 million a year. Why would Microsoft incur that cost when Chromium is available under a license that allows Microsoft to do whatever it wants with it?
      - Eggpants 11mo ago
        You could say the same thing about all Microsoft products then. How many full time developers does it take to support Windows 11 when Linux is available, SqlServer when Postgres is available, Office when LibreOffice exists? And so on all under licenses that allows Microsoft do whatever it wants with? They should be embarrassed to do better, not spin it into a “wise business move” aka transfer that money into executive bonuses.
        hollerith 11mo ago
        Microsoft gets a lot of its revenue from the sale of licenses and subscriptions for Windows and Office. An unreliable source that gives fast answers to questions tells me that the segments responsible for those two softwares have revenue of about $13 and about 20 billion per quarter respectively. In contrast, basically no one derives any significant revenue from the sale of licenses or subscriptions for web browsers. As long as Microsoft can modify Chromium to have Microsoft's branding, to nag the user into using Microsoft Copilot and to direct search queries to Bing instead of Google Search, why should Microsoft care about web browsers? It gets worse. Any browser Microsoft offers needs to work well on almost any web site. These web sites (of which there are 100s of 1000s) in turn are maintained by developers (hi, web devs!) that tend to be eager to embrace any new technology Google puts into Chrome, with the result that Microsoft must responding by putting the same technological capabilities into its own web browser. Note that the same does not hold for Windows: there is no competitor to Microsoft offering a competitor to Windows that is constantly inducing the maintainers of Windows applications to embrace new technologies, requiring Microsoft to incur the expense of applying engineering pressure to Windows to keep up. This suggests to me that maintaining Windows is actually significantly cheaper than it would be to maintain an independent mainstream browser. An independent mainstream browser is probably the most expensive category of software to create and to maintain excepting only foundational AI models. "Independent" here means "not a fork of Chromium or Firefox". "Mainstream" means "capable of correctly rendering the vast majority of web sites a typical person might want to visit".
danielscrubs 11mo ago
Well put. I haven’t thought about it like that.
musebox35 11mo ago
I have deep respect for cuda and Nvidia engineering. However, the arguments above seem to totally ignore Google Search indexing and query software stack. They are the king of distributed software and also hardware that scales. That is way TPUs are a thing now and they can compute with Nvidia where AMD failed. Distributed software is the bread and butter of Google with their multi-decade investment from day zero out of necessity. When you have to update an index of an evolving set of billions of documents daily and do that online while keeping subsecond query capability across the globe, that should teach you a few things about deep software stacks.
bluelightning2k 11mo ago
You're suggesting Waymo isn't deep software? Or Tensorflow? Or Android? The Go programming language? Or MapReduce, AlphaGo, Kubernetes, the transformer, Chrome/Chromium or Gvisor? You must have an amazing CV to think these are shallow projects.
- qdotme 11mo ago
  No, I just realize these for what they are - reasonable projects at the exploitation (rather than exploration) stage of any industry. I’d say I have an average CV in the EECS world, but also relatively humble perspective of what is and isn’t bleeding edge. And as the industry expands, the volume „inside” the bleeding edge is exploitation, while the surface is the exploration. Waymo? Maybe; but that’s acquisition and they haven’t done much deep work since. Tensorflow is a handy and very useful DSL, but one that is shallow (builds heavily on CUDA and TPUs etc); Android is another acquisition, and rather incremental growth since; Go is a nth C-like language (so neither Dennis Richie nor Bjarne Stroustrup level work); MapReduce is a darn common concept in HPC (SGI had libraries for it in the 1990s) and implementation was pretty average. AlphaGo - another acquisition, and not much deep work since; Kubernetes is a layer over Linux Namespaces to solve - well - shallow and broad problems; Chrome/Chromium is the 4th major browser that reached dominance and essentially anyone with a 1B to spare can build one.. gVisor is another thin, shallow layer. What I mean by deep software, is a product that requires 5-10y of work before it is useful, that touches multiple layers of software stack (ideally all from hardware to application) etc. But these types of jobs are relatively rare in the 2020s software world (pretty common in robotics and new space) - they were common in the 1990s where I got my calibration values ;) Netscape and Palm Pilot was a „whoa”. Chromium and Android are evolutions.
  - SR2Z 11mo ago
    > No, I just realize these for what they are - reasonable projects at the exploitation (rather than exploration) stage of any industry. I get that bashing on Google is fun, but TensorFlow was the FIRST modern end-user ML library. JAX, an optimizing backend for it, is in its own league even today. The damn thing is almost ten years old already! Waymo is literally the only truly publicly available robotaxi company. I don't know where you get the idea that it's an acquisition; it's the spun-off incarnation of the Google self-driving car project that for years was the butt of "haha, software engineers think they're real engineers" jokes. Again, more than a decade of development on this. Kubernetes is a refinement of Borg, which Google was using to do containerized workloads all the way back in 2003! How's that not a deep project?
dumah 11mo ago
These companies innovate in all of those areas and direct those resources towards building hyper-scale custom infrastructure, including CPU, TPU, GPU, and custom networking hardware for the largest cloud systems, and conduct research and development on new compilers and operating system components to exploit them. They're building it for themselves and employ world-class experts across the entire stack. How can NVIDIA develop "more integrated" solutions when they are primarily building for these companies, as well as many others? Examples of these companies doing things you mention as being somehow unique to or characteristic of NVIDIA: Complex kernel drivers or modules: - AWS: Nitro, ENA/EFA, Firecracker, NKI, bottlerocket - Google: gasket/apex, gve, binder - Meta: Katran, bpfilter, cgroup2, oomd, btrfs Hardware simulators: - AWS: Neuron, Annapurna builds simulations for nitro, graviton, inferentia and validates aws instances built for EDA services - Google: Goldfish, Ranchu, Cuttlefish - Meta: Arcadia, MTIA, CFD for thermal management Optimizing Compilers: - Amazon: NNVM, Neo-AI - Google: MLIR, XLA, IREE - Meta: Glow, Triton, LLM Compiler Acceleration Libraries: - Amazon: NeuronX, aws-ofi-nccl - Google: Jax, TF - Meta: FBGEMM, QNNPACK

whywhywhywhy 11mo ago

If CUDA were as trivial to replicate as you say then Nvidia wouldn’t be what it is today.

whatevaa 11mo ago
CUDA is not hard to replicate, but the network effects make it very hard to break trough with new product. Just like with everything when network effeft applies.

Miraste 11mo ago

Meta makes websites and apps. Historically, they haven't succeeded at lower-level development. A somewhat recent example was when they tried to make a custom OS for their VR headsets, completely failed, and had to continue using Android.

coredog64 11mo ago
Remind me which company originated PyTorch?
- kanbankaren 11mo ago
  Remind me that PyTorch is not a GPU driver.
dumah 11mo ago
You're generalizing a failure at delivering one consumer solution and ignoring the successful infrastructure research and development that occurs behind the scenes. Meta builds hardware from chip to cluster to datacenter scale, and drives research into simulation at every scale, all the way to CFD simulation of datacenter thermal management.
- Miraste 11mo ago
  More than one failure. They had a project to make a custom chip for model training a few years ago, and they scrapped it. Now they have another one, which entered testing in March. I don't think it's going well, because testing should have wrapped up recently, right before the news that they're in serious talks to buy a lot of TPUs from Google. On the other side of the stack, Llama 4 was a disaster and they haven't shipped anything since. They have the money and talent to do it. As you point out, they do have major successes in areas that take real engineering. But they also have a lot of failures. It will depend how the internal politics play out, I imagine.