4 ms·

> Customer support. I've never seen so many people get up to speed so fast on something. It's unlike anything ever. Are you saying that you've seen enterprises

> Customer support. I've never seen so many people get up to speed so fast on something. It's unlike anything ever. Are you saying that you've seen enterprises developing, training and running their own in-house LLMs from scratch directly on large (i.e. 100s to 1000s) GPU clusters, whether on-prem or cloud, for this to be relevant to Nvidia? Pardon my skepticism but it seems odd that a generic Fortune 500 co has the in-house talent and will-power to manage large distributed training runs when much easier and cheaper alternatives like OpenAI/open-source models or one of the Google/MS/AWS MLaaS options are available. > With the relevant audience, Nvidia is the brand which is the strongest. Google could just drop support for TPU, AMD isn't viewed as currently credible for people doing the work, nor Intel. I think we're confusing some things here. Right now, there is no good alternative for anyone requiring H100s for loyalty to even matter, this could very easily change with the next generation of accelerator chips. Intel had the strongest CPU "brand" for a while and enterprises/datacenters readily switched to AMD when it became the better option. > You may be right, only time will tell. I've been involved with GPU use for general purpose workloads since Cuda was launched. At every step of the way there was apparently credible competition at different layers of the stack just around the corner. That's over 15 years. OpenCL, ASICs, FPGA, Intel this that and the other, AMD this that and the other. TPU. Others I've forgotten. The TAM, and profit margin, for enterprise-grade GPUs (or accelerators) is several orders of magnitude larger than it has ever been including the crypto craze.

danielmarkbruce 3y ago
You seem to be moving the goalposts. First it was using in production, now it's building foundation models.. But, to answer: Not foundation models (although, Bloomberg did that for another use case). Fine tuned versions of falcon, now llama 2, open assistant and others. They are running both the fine tuning (not that it's much) and inference on A100s, inferentia and TPU. The skillset required to fine tune a model for a customer support use case is night and day from building a (good) foundation model. Right now it doesn't matter, but you are suggesting there will be real competition soon. I'm saying, it will matter. Intel did have a great brand. And they proceeded to screw up and it caught up with them, but it took a long time. A company with a moat can destroy it with time and effort. It is huge market. And it's a huge lift to build a product that can compete with Nvidia at this point. I'd handicap it at 2-1 against someone being a real competitor within a decade.
- haldujai 3y ago
  > You seem to be moving the goalposts. First it was using in production, now it's building foundation models.. I'm not moving the goal posts but maybe I wasn't clear. When we talk about corporations adopting fine-tuning and inference that's not especially relevant to H100 sales which is the main cash-cow (~10B of revenue at 80-90% margins) and Nvidia's massive market cap growth. What is relevant is corporations like Inflection AI building 22k H100 clusters. I work in academia so PyTorch is more common but are people in industry fine-tuning LLMs actually working directly with CUDA that much for this to be a big moat?
  - danielmarkbruce 3y ago
    Companies are using both A100 and H100 for inference. The datacenter numbers include both, and I don't believe they break it down further. And no, by and large, large enterprises are not building out large clusters - but they are much of the demand for all those cloud provider build outs. No, no one is using CUDA directly. But if you are a vendor with a new equivalent, it's no small feat integrating it with a framework like PyTorch. There is a lot of work required by various parties.
    - haldujai 3y ago
      It would be interesting to see what the breakdown is on H100/A100 users. I would expect most inference users are similar to my lab and max out at a DGX node rather than being the bulk of users. PyTorch has gotten a lot better on TPUs this year, I don’t believe there’s much of a performance hit now. Jax and TF (I don’t use the latter anymore) of course work. I never used Gaudi2 but it apparently works. All of this to say, is it possible that Intel gets their fabs working and strategically partners with their long-time partner MS and OpenAI to extend Triton to Gaudi3 or 4 and be a threat to Nvidia within 2-3 years? Absolutely. Is it similarly possible that Google increases development on Jax and TPUv5? Sure. Neither of these possibilities, regardless if you think they’re probable or improbable, would need a decade to catch up to Nvidia.
      - 3y ago

danielmarkbruce 3y ago

You seem to be moving the goalposts. First it was using in production, now it's building foundation models.. But, to answer: Not foundation models (although, Bloomberg did that for another use case). Fine tuned versions of falcon, now llama 2, open assistant and others. They are running both the fine tuning (not that it's much) and inference on A100s, inferentia and TPU. The skillset required to fine tune a model for a customer support use case is night and day from building a (good) foundation model. Right now it doesn't matter, but you are suggesting there will be real competition soon. I'm saying, it will matter. Intel did have a great brand. And they proceeded to screw up and it caught up with them, but it took a long time. A company with a moat can destroy it with time and effort. It is huge market. And it's a huge lift to build a product that can compete with Nvidia at this point. I'd handicap it at 2-1 against someone being a real competitor within a decade.

haldujai 3y ago
> You seem to be moving the goalposts. First it was using in production, now it's building foundation models.. I'm not moving the goal posts but maybe I wasn't clear. When we talk about corporations adopting fine-tuning and inference that's not especially relevant to H100 sales which is the main cash-cow (~10B of revenue at 80-90% margins) and Nvidia's massive market cap growth. What is relevant is corporations like Inflection AI building 22k H100 clusters. I work in academia so PyTorch is more common but are people in industry fine-tuning LLMs actually working directly with CUDA that much for this to be a big moat?
- danielmarkbruce 3y ago
  Companies are using both A100 and H100 for inference. The datacenter numbers include both, and I don't believe they break it down further. And no, by and large, large enterprises are not building out large clusters - but they are much of the demand for all those cloud provider build outs. No, no one is using CUDA directly. But if you are a vendor with a new equivalent, it's no small feat integrating it with a framework like PyTorch. There is a lot of work required by various parties.
  - haldujai 3y ago
    It would be interesting to see what the breakdown is on H100/A100 users. I would expect most inference users are similar to my lab and max out at a DGX node rather than being the bulk of users. PyTorch has gotten a lot better on TPUs this year, I don’t believe there’s much of a performance hit now. Jax and TF (I don’t use the latter anymore) of course work. I never used Gaudi2 but it apparently works. All of this to say, is it possible that Intel gets their fabs working and strategically partners with their long-time partner MS and OpenAI to extend Triton to Gaudi3 or 4 and be a threat to Nvidia within 2-3 years? Absolutely. Is it similarly possible that Google increases development on Jax and TPUv5? Sure. Neither of these possibilities, regardless if you think they’re probable or improbable, would need a decade to catch up to Nvidia.
    - 3y ago