14 ms·

It was only a matter of time. Google announced theirs years ago, Amazon announced theirs last year. Right now NVIDIA has the lead because they have the better

It was only a matter of time. Google announced theirs years ago, Amazon announced theirs last year. Right now NVIDIA has the lead because they have the better software, but they can't make the chips fast enough. Will be interesting to see if their better software continues to keep them in the lead or if people are more interested in getting the capacity in any form.

rehitman 3y ago
I would say software is very important. We already have tons of different models, standards, library, etc. I usually have smooth experience if I am using NVDA, but any other variation, I have to spend some times to get things started. Supporting the common librarires that I use is very important for me to chose the cloud platform.
Jagerbizzle 3y ago
Considering that Jensen is on stage with Satya at the moment sharing the keynote of Microsoft Ignite, I suspect NVIDIA won't be going anywhere anytime soon.
- paulpan 3y ago
  A bit surprised to see Jensen's stage appearance since clearly Microsoft's success with its own AI chips means less business for Nvidia's chips. Because different than the ARM chip also announced in the same Ignite event, Microsoft doesn't exactly "need" nor can fully utilize an AI chip. Google trains its foundational models (e.g. Gemini) on its own TPU hardware but Microsoft's is heavily reliant on OpenAI for its generative AI serving needs. Unless Microsoft is planning to acquire OpenAI fully and switch over from Nvidia hardware...
  - echelon 3y ago
    > Unless Microsoft is planning to acquire OpenAI fully They're going to play a modified version of the old Rareware trick. It's also a pretty great game to buy up OpenAI equity, which ultimately gets spent on Microsoft compute. Two birds, one stone.
    - WanderPanda 3y ago
      I don’t think the margins here are so considerable that we can assume revenue = profit
      - echelon 3y ago
        They can be losing money, but gaining in market share and moat. Almost nobody in this game cares about profit right now.
    - kmeisthax 3y ago
      ...they're going to overbid on a studio that was actively falling apart, after being rebuffed from buying one of the biggest giants in the business[0], all as part of an ill-advised attempt to muscle into a game business they didn't understand? [0] Microsoft tried to buy Nintendo very early on
  - aseipp 3y ago
    Microsoft absolutely runs their own models on their own hardware, at scale, and they have done so for years just like every other hyperscaler -- Project Brainwave was first publicly talked about as far back as 2018. The generative LLM craze is a recent phenomenon in comparison. They are absolutely going to go all in on putting AI functionality in Bing, in Excel, in Windows, etc etc. To do that, you need hardware. None of this is really strange. It also wasn't strange when Google announced H100 systems while also pushing TPUs they developed. Microsoft has Jensen on stage because customers of Microsoft Azure demand Nvidia products. Customers of Google Cloud demand Nvidia products. So, they provide them those products, because not providing them loses those customers. It's that simple. Everyone involved in these deals acknowledges this.
    - buildbot 3y ago
      Awesome! Someone who knows about Brainwave!
      - aseipp 3y ago
        Yeah! Did/do you work on it? The original publications were good timing; I was working as a consultant on an FPGA-based ML accelerator at the time the original stuff was talked about, and I really enjoyed reading everything I could about Brainwave! Really neat project from both a system design perspective (e.g. heterogeneous compiler) to the choice of using and interconnecting FPGAs and integrating the network/software/ML stack (IIRC, there was a good paper on the overlay network they used to make those custom functions available on the global network fabric.) I'm guessing at this point the ASICs make a lot more economic sense, though. :)
        buildbot 3y ago
        Sorta! Tangentially on the data science side - Brainwave pre-dates me being at MS though. It's a really cool project and I'm glad someone else also thinks so :) I was enamored with FPGAs from undergrad, one of the professors I did research with was a consultant on the original Catapult project, I thought it was the coolest idea. It was pretty awesome to be able to come full circle a fews years later.
rapsey 3y ago
nVidia still has the monopoly on training. Everyone else is just making chips for inference.
- muro 3y ago
  https://cloud.google.com/tpu/docs/training-on-tpu-pods https://cloud.google.com/tpu/docs/training-on-tpu-pods
- adrian_b 3y ago
  While NVIDIA provides the best absolute performance for training, Intel (i.e. Gaudi) already provides much better training performance per dollar. The funny thing is that this fact has been shown inadvertently by NVIDIA: https://www.servethehome.com/nvidia-shows-intel-gaudi2-is-4x-better-performance-per-dollar-than-its-h100/ https://www.servethehome.com/nvidia-shows-intel-gaudi2-is-4x...
  - sgillen 3y ago
    That article shows that it takes about 50x as long to train gpt-3 with intel's offering vs Nvidia. At least in the current environment, if you are training llms I think almost no amount of cost savings can justify that.
    - adrian_b 3y ago
      That 50X is only if you can afford one thousand NVIDIA H100. There cannot be more than a handful of companies in the entire world that could afford such a huge price (tens of millions of $). In comparison with a still extremely expensive cluster of 64 NVIDIA H100, the difference in speed would reduce to only two to three times, and paying several times less for the entire training becomes very attractive.
      - lazide 3y ago
        You’ve got a really warped sense of scale if you think only a handful of companies couldn’t afford tens of millions of dollars. That’s complete pocket change for any of the Fortune 500.
belval 3y ago
> Amazon announced theirs last year Inferentia (inf1) was GA'ed in December 2019 so it's actually almost 4 years old now. The trainium (trn1) chips and the Inferentia 2 (inf2) refresh is indeed 1 year old though.
- jedberg 3y ago
  I was referring to trainium.
- ShamelessC 3y ago
  Yikes those names are horrible.
  - belval 3y ago
    Why? Inferentia => inference, trainium => training. Given the usually naming of AWS product, having one where the name roughly matches what it does is pretty good? TPU is pretty good but is associated with Google. MTIA is an acronym but still maps to what the chip does. ~~"Cobalt" is worse as it does not mean anything~~ . Cobalt is the CPU chip, MAIA is the accelerator so this matches Meta's naming.
    - ShamelessC 3y ago
      > Why? Inferentia => inference, trainium => training. Funny, that's precisely why I think the names are bad. It's like if Google had chosen "Search-ola" as their name. Way too on the nose and/or lazy. Having said that, I don't really care all that much and I imagine that may have been the spirit of those who chose the names.
      - mdaniel 3y ago
        heh, as someone who has to deal with this nonsense all day <https://aws.amazon.com/products/ https://aws.amazon.com/products/> I would for sure welcome some straightforward naming. $(echo "AWS Fargate" | sed s/Fargate/ServerlessContainerium/)
scarface_74 3y ago
If they “can’t make the chips fast enough” being TSMC’s second highest volume customer behind Apple and probably second in priority, what chance does Microsoft have getting enough of TSMCs capacity?
- donatzsky 3y ago
  Microsoft does have the benefit of not having any customers other than themselves, so volumes are smaller.
- tambre 3y ago
  They're more constrained by advanced packaging (CoWoS) capacity rather than the manufacturing of the silicon.
  - tyfighter 3y ago
    But from the pictures, the Maia 100 is also using CoWoS packaging. That will be necessary for any new HBM chip.
  - localhost 3y ago
    The latest Stratechery interview had Pat Gelsinger on it and he was talking about how Intel is happy to package up TSMC dies in their foundries. Relevant quote: > So it really brings together many of the things that Intel is doing as an IDM, now bringing it together in a heterogeneous environment where we’re taking TSMC dies. We’re going to be using other foundries in the industry, we’re standardizing that with UCIe. So I really see ourself as the front end of this multi-chip chiplet world doing so in the Intel way, standardizing it for the industry’s participation with UCIE, and then just winning a better technology.
- tw04 3y ago
  > what chance does Microsoft have getting enough of TSMCs capacity? Looking at their balance sheet: plenty. TSMC is all about pay to play. Apple is first in line because they’re willing to pay to be first in line. I have no doubt Microsoft can justify spending the money they save not buying nvidia into getting some priority access from TSMC. Also keep in mind Apple is now on 3nm so there’s likely spare 5nm.
sofixa 3y ago
> Amazon announced theirs last year Amazon has had Graviton chips (after they acquired Annapurna Labs) since 2018. Or do you mean specifically AI-oriented chips?

rehitman 3y ago

I would say software is very important. We already have tons of different models, standards, library, etc. I usually have smooth experience if I am using NVDA, but any other variation, I have to spend some times to get things started. Supporting the common librarires that I use is very important for me to chose the cloud platform.

Jagerbizzle 3y ago

Considering that Jensen is on stage with Satya at the moment sharing the keynote of Microsoft Ignite, I suspect NVIDIA won't be going anywhere anytime soon.

paulpan 3y ago
A bit surprised to see Jensen's stage appearance since clearly Microsoft's success with its own AI chips means less business for Nvidia's chips. Because different than the ARM chip also announced in the same Ignite event, Microsoft doesn't exactly "need" nor can fully utilize an AI chip. Google trains its foundational models (e.g. Gemini) on its own TPU hardware but Microsoft's is heavily reliant on OpenAI for its generative AI serving needs. Unless Microsoft is planning to acquire OpenAI fully and switch over from Nvidia hardware...
- echelon 3y ago
  > Unless Microsoft is planning to acquire OpenAI fully They're going to play a modified version of the old Rareware trick. It's also a pretty great game to buy up OpenAI equity, which ultimately gets spent on Microsoft compute. Two birds, one stone.
  - WanderPanda 3y ago
    I don’t think the margins here are so considerable that we can assume revenue = profit
    - echelon 3y ago
      They can be losing money, but gaining in market share and moat. Almost nobody in this game cares about profit right now.
  - kmeisthax 3y ago
    ...they're going to overbid on a studio that was actively falling apart, after being rebuffed from buying one of the biggest giants in the business[0], all as part of an ill-advised attempt to muscle into a game business they didn't understand? [0] Microsoft tried to buy Nintendo very early on
- aseipp 3y ago
  Microsoft absolutely runs their own models on their own hardware, at scale, and they have done so for years just like every other hyperscaler -- Project Brainwave was first publicly talked about as far back as 2018. The generative LLM craze is a recent phenomenon in comparison. They are absolutely going to go all in on putting AI functionality in Bing, in Excel, in Windows, etc etc. To do that, you need hardware. None of this is really strange. It also wasn't strange when Google announced H100 systems while also pushing TPUs they developed. Microsoft has Jensen on stage because customers of Microsoft Azure demand Nvidia products. Customers of Google Cloud demand Nvidia products. So, they provide them those products, because not providing them loses those customers. It's that simple. Everyone involved in these deals acknowledges this.
  - buildbot 3y ago
    Awesome! Someone who knows about Brainwave!
    - aseipp 3y ago
      Yeah! Did/do you work on it? The original publications were good timing; I was working as a consultant on an FPGA-based ML accelerator at the time the original stuff was talked about, and I really enjoyed reading everything I could about Brainwave! Really neat project from both a system design perspective (e.g. heterogeneous compiler) to the choice of using and interconnecting FPGAs and integrating the network/software/ML stack (IIRC, there was a good paper on the overlay network they used to make those custom functions available on the global network fabric.) I'm guessing at this point the ASICs make a lot more economic sense, though. :)
      - buildbot 3y ago
        Sorta! Tangentially on the data science side - Brainwave pre-dates me being at MS though. It's a really cool project and I'm glad someone else also thinks so :) I was enamored with FPGAs from undergrad, one of the professors I did research with was a consultant on the original Catapult project, I thought it was the coolest idea. It was pretty awesome to be able to come full circle a fews years later.

rapsey 3y ago

nVidia still has the monopoly on training. Everyone else is just making chips for inference.

muro 3y ago
https://cloud.google.com/tpu/docs/training-on-tpu-pods https://cloud.google.com/tpu/docs/training-on-tpu-pods
adrian_b 3y ago
While NVIDIA provides the best absolute performance for training, Intel (i.e. Gaudi) already provides much better training performance per dollar. The funny thing is that this fact has been shown inadvertently by NVIDIA: https://www.servethehome.com/nvidia-shows-intel-gaudi2-is-4x-better-performance-per-dollar-than-its-h100/ https://www.servethehome.com/nvidia-shows-intel-gaudi2-is-4x...
- sgillen 3y ago
  That article shows that it takes about 50x as long to train gpt-3 with intel's offering vs Nvidia. At least in the current environment, if you are training llms I think almost no amount of cost savings can justify that.
  - adrian_b 3y ago
    That 50X is only if you can afford one thousand NVIDIA H100. There cannot be more than a handful of companies in the entire world that could afford such a huge price (tens of millions of $). In comparison with a still extremely expensive cluster of 64 NVIDIA H100, the difference in speed would reduce to only two to three times, and paying several times less for the entire training becomes very attractive.
    - lazide 3y ago
      You’ve got a really warped sense of scale if you think only a handful of companies couldn’t afford tens of millions of dollars. That’s complete pocket change for any of the Fortune 500.

belval 3y ago

> Amazon announced theirs last year Inferentia (inf1) was GA'ed in December 2019 so it's actually almost 4 years old now. The trainium (trn1) chips and the Inferentia 2 (inf2) refresh is indeed 1 year old though.

jedberg 3y ago
I was referring to trainium.
ShamelessC 3y ago
Yikes those names are horrible.
- belval 3y ago
  Why? Inferentia => inference, trainium => training. Given the usually naming of AWS product, having one where the name roughly matches what it does is pretty good? TPU is pretty good but is associated with Google. MTIA is an acronym but still maps to what the chip does. ~~"Cobalt" is worse as it does not mean anything~~ . Cobalt is the CPU chip, MAIA is the accelerator so this matches Meta's naming.
  - ShamelessC 3y ago
    > Why? Inferentia => inference, trainium => training. Funny, that's precisely why I think the names are bad. It's like if Google had chosen "Search-ola" as their name. Way too on the nose and/or lazy. Having said that, I don't really care all that much and I imagine that may have been the spirit of those who chose the names.
    - mdaniel 3y ago
      heh, as someone who has to deal with this nonsense all day <https://aws.amazon.com/products/ https://aws.amazon.com/products/> I would for sure welcome some straightforward naming. $(echo "AWS Fargate" | sed s/Fargate/ServerlessContainerium/)

scarface_74 3y ago

If they “can’t make the chips fast enough” being TSMC’s second highest volume customer behind Apple and probably second in priority, what chance does Microsoft have getting enough of TSMCs capacity?

donatzsky 3y ago
Microsoft does have the benefit of not having any customers other than themselves, so volumes are smaller.
tambre 3y ago
They're more constrained by advanced packaging (CoWoS) capacity rather than the manufacturing of the silicon.
- tyfighter 3y ago
  But from the pictures, the Maia 100 is also using CoWoS packaging. That will be necessary for any new HBM chip.
- localhost 3y ago
  The latest Stratechery interview had Pat Gelsinger on it and he was talking about how Intel is happy to package up TSMC dies in their foundries. Relevant quote: > So it really brings together many of the things that Intel is doing as an IDM, now bringing it together in a heterogeneous environment where we’re taking TSMC dies. We’re going to be using other foundries in the industry, we’re standardizing that with UCIe. So I really see ourself as the front end of this multi-chip chiplet world doing so in the Intel way, standardizing it for the industry’s participation with UCIE, and then just winning a better technology.
tw04 3y ago
> what chance does Microsoft have getting enough of TSMCs capacity? Looking at their balance sheet: plenty. TSMC is all about pay to play. Apple is first in line because they’re willing to pay to be first in line. I have no doubt Microsoft can justify spending the money they save not buying nvidia into getting some priority access from TSMC. Also keep in mind Apple is now on 3nm so there’s likely spare 5nm.

sofixa 3y ago

> Amazon announced theirs last year Amazon has had Graviton chips (after they acquired Annapurna Labs) since 2018. Or do you mean specifically AI-oriented chips?