4 ms·

What is full steam ahead for enterprises? It's not like they're throwing autoregressive LLMs into production any time soon. In any case Nvidia is expecting to

What is full steam ahead for enterprises? It's not like they're throwing autoregressive LLMs into production any time soon. In any case Nvidia is expecting to ship ~550k H100s in 2023, hardly enough to satisfy every user. Tesla decided to in-house. TPUv4 and Gaudi2 exceeded A100 performance, they just never hit scale or the market and then Hopper added optimization for transformers rendering these chips relatively obsolete. Nvidia's lead is not unassailable and it seems incredibly unlikely that they would not face serious competition within the next 2-3 years given the $ being thrown around.

danielmarkbruce 3y ago
Large enterprises are already putting them into production. I have direct experience with it. It's not unassailable. But it's going to take a lot to make any difference to Nvidia's volume or pricing, let alone a meaningful difference. They already face serious competitors in google and aws with TPU and inferentia, but those competitors are at a pretty big disadvantage for now (and others too). The cuda ecosystem is a big advantage. Nvidia has a lot of leverage with semi manufacturers because of volume. They spend way more on chip R&D than their competitors in the space. They have brand recognition. You can buy and own Nvidia chips v tpu and inferentia. It's... a tough road ahead for competitors.
- haldujai 3y ago
  > Large enterprises are already putting them into production. I have direct experience with it. For what function have you experienced LLMs being used at significant scale in production right now? It seems unlikely most enterprises have built up sufficient technical know-how to run these workloads in-house already. > They already face serious competitors in google and aws with TPU and inferentia TPUv4s aren't widely available and are not competitive with Hopper. Inferentia isn't for training and is also not competitive on throughput. > The cuda ecosystem is a big advantage. Is it really though? A year ago it wasn't the case (see TPUs). Nvidia hasn't maintained a stranglehold on the software stack. > Nvidia has a lot of leverage with semi manufacturers because of volume. With TSMC*. If we believe Pat Gelsinger, Intel is starting to catch-up and their roadmap would place them ahead of TSMC in H2 2024. > They have brand recognition. You can buy and own Nvidia chips v tpu and inferentia. Intel, AMD and Google don't have brand recognition? There isn't a competitive alternative today for sale or rent, that doesn't mean this can't/won't change. It also doesn't mean they won't become available for enterprise purchase when we're talking about 60B/year in revenue. > It's... a tough road ahead for competitors. While hard I disagree that their moat is as wide as being touted. At the end of the day, whoever gets the most FLOPS, can deliver product faster and/or is cheaper will win as long as it's trivial to migrate workloads. I really don't believe anyone actually cares who the manufacturer is.
  - danielmarkbruce 3y ago
    Customer support. I've never seen so many people get up to speed so fast on something. It's unlike anything ever. Yep. I believe so. I don't believe Pat. With the relevant audience, Nvidia is the brand which is the strongest. Google could just drop support for TPU, AMD isn't viewed as currently credible for people doing the work, nor Intel. You may be right, only time will tell. I've been involved with GPU use for general purpose workloads since Cuda was launched. At every step of the way there was apparently credible competition at different layers of the stack just around the corner. That's over 15 years. OpenCL, ASICs, FPGA, Intel this that and the other, AMD this that and the other. TPU. Others I've forgotten.
    - haldujai 3y ago
      > Customer support. I've never seen so many people get up to speed so fast on something. It's unlike anything ever. Are you saying that you've seen enterprises developing, training and running their own in-house LLMs from scratch directly on large (i.e. 100s to 1000s) GPU clusters, whether on-prem or cloud, for this to be relevant to Nvidia? Pardon my skepticism but it seems odd that a generic Fortune 500 co has the in-house talent and will-power to manage large distributed training runs when much easier and cheaper alternatives like OpenAI/open-source models or one of the Google/MS/AWS MLaaS options are available. > With the relevant audience, Nvidia is the brand which is the strongest. Google could just drop support for TPU, AMD isn't viewed as currently credible for people doing the work, nor Intel. I think we're confusing some things here. Right now, there is no good alternative for anyone requiring H100s for loyalty to even matter, this could very easily change with the next generation of accelerator chips. Intel had the strongest CPU "brand" for a while and enterprises/datacenters readily switched to AMD when it became the better option. > You may be right, only time will tell. I've been involved with GPU use for general purpose workloads since Cuda was launched. At every step of the way there was apparently credible competition at different layers of the stack just around the corner. That's over 15 years. OpenCL, ASICs, FPGA, Intel this that and the other, AMD this that and the other. TPU. Others I've forgotten. The TAM, and profit margin, for enterprise-grade GPUs (or accelerators) is several orders of magnitude larger than it has ever been including the crypto craze.

danielmarkbruce 3y ago

Large enterprises are already putting them into production. I have direct experience with it. It's not unassailable. But it's going to take a lot to make any difference to Nvidia's volume or pricing, let alone a meaningful difference. They already face serious competitors in google and aws with TPU and inferentia, but those competitors are at a pretty big disadvantage for now (and others too). The cuda ecosystem is a big advantage. Nvidia has a lot of leverage with semi manufacturers because of volume. They spend way more on chip R&D than their competitors in the space. They have brand recognition. You can buy and own Nvidia chips v tpu and inferentia. It's... a tough road ahead for competitors.

haldujai 3y ago
> Large enterprises are already putting them into production. I have direct experience with it. For what function have you experienced LLMs being used at significant scale in production right now? It seems unlikely most enterprises have built up sufficient technical know-how to run these workloads in-house already. > They already face serious competitors in google and aws with TPU and inferentia TPUv4s aren't widely available and are not competitive with Hopper. Inferentia isn't for training and is also not competitive on throughput. > The cuda ecosystem is a big advantage. Is it really though? A year ago it wasn't the case (see TPUs). Nvidia hasn't maintained a stranglehold on the software stack. > Nvidia has a lot of leverage with semi manufacturers because of volume. With TSMC*. If we believe Pat Gelsinger, Intel is starting to catch-up and their roadmap would place them ahead of TSMC in H2 2024. > They have brand recognition. You can buy and own Nvidia chips v tpu and inferentia. Intel, AMD and Google don't have brand recognition? There isn't a competitive alternative today for sale or rent, that doesn't mean this can't/won't change. It also doesn't mean they won't become available for enterprise purchase when we're talking about 60B/year in revenue. > It's... a tough road ahead for competitors. While hard I disagree that their moat is as wide as being touted. At the end of the day, whoever gets the most FLOPS, can deliver product faster and/or is cheaper will win as long as it's trivial to migrate workloads. I really don't believe anyone actually cares who the manufacturer is.
- danielmarkbruce 3y ago
  Customer support. I've never seen so many people get up to speed so fast on something. It's unlike anything ever. Yep. I believe so. I don't believe Pat. With the relevant audience, Nvidia is the brand which is the strongest. Google could just drop support for TPU, AMD isn't viewed as currently credible for people doing the work, nor Intel. You may be right, only time will tell. I've been involved with GPU use for general purpose workloads since Cuda was launched. At every step of the way there was apparently credible competition at different layers of the stack just around the corner. That's over 15 years. OpenCL, ASICs, FPGA, Intel this that and the other, AMD this that and the other. TPU. Others I've forgotten.
  - haldujai 3y ago
    > Customer support. I've never seen so many people get up to speed so fast on something. It's unlike anything ever. Are you saying that you've seen enterprises developing, training and running their own in-house LLMs from scratch directly on large (i.e. 100s to 1000s) GPU clusters, whether on-prem or cloud, for this to be relevant to Nvidia? Pardon my skepticism but it seems odd that a generic Fortune 500 co has the in-house talent and will-power to manage large distributed training runs when much easier and cheaper alternatives like OpenAI/open-source models or one of the Google/MS/AWS MLaaS options are available. > With the relevant audience, Nvidia is the brand which is the strongest. Google could just drop support for TPU, AMD isn't viewed as currently credible for people doing the work, nor Intel. I think we're confusing some things here. Right now, there is no good alternative for anyone requiring H100s for loyalty to even matter, this could very easily change with the next generation of accelerator chips. Intel had the strongest CPU "brand" for a while and enterprises/datacenters readily switched to AMD when it became the better option. > You may be right, only time will tell. I've been involved with GPU use for general purpose workloads since Cuda was launched. At every step of the way there was apparently credible competition at different layers of the stack just around the corner. That's over 15 years. OpenCL, ASICs, FPGA, Intel this that and the other, AMD this that and the other. TPU. Others I've forgotten. The TAM, and profit margin, for enterprise-grade GPUs (or accelerators) is several orders of magnitude larger than it has ever been including the crypto craze.