3 ms·

You seem to be moving the goalposts. First it was using in production, now it's building foundation models.. But, to answer: Not foundation models (although, Bl

You seem to be moving the goalposts. First it was using in production, now it's building foundation models.. But, to answer: Not foundation models (although, Bloomberg did that for another use case). Fine tuned versions of falcon, now llama 2, open assistant and others. They are running both the fine tuning (not that it's much) and inference on A100s, inferentia and TPU. The skillset required to fine tune a model for a customer support use case is night and day from building a (good) foundation model. Right now it doesn't matter, but you are suggesting there will be real competition soon. I'm saying, it will matter. Intel did have a great brand. And they proceeded to screw up and it caught up with them, but it took a long time. A company with a moat can destroy it with time and effort. It is huge market. And it's a huge lift to build a product that can compete with Nvidia at this point. I'd handicap it at 2-1 against someone being a real competitor within a decade.

haldujai 3y ago
> You seem to be moving the goalposts. First it was using in production, now it's building foundation models.. I'm not moving the goal posts but maybe I wasn't clear. When we talk about corporations adopting fine-tuning and inference that's not especially relevant to H100 sales which is the main cash-cow (~10B of revenue at 80-90% margins) and Nvidia's massive market cap growth. What is relevant is corporations like Inflection AI building 22k H100 clusters. I work in academia so PyTorch is more common but are people in industry fine-tuning LLMs actually working directly with CUDA that much for this to be a big moat?
- danielmarkbruce 3y ago
  Companies are using both A100 and H100 for inference. The datacenter numbers include both, and I don't believe they break it down further. And no, by and large, large enterprises are not building out large clusters - but they are much of the demand for all those cloud provider build outs. No, no one is using CUDA directly. But if you are a vendor with a new equivalent, it's no small feat integrating it with a framework like PyTorch. There is a lot of work required by various parties.
  - haldujai 3y ago
    It would be interesting to see what the breakdown is on H100/A100 users. I would expect most inference users are similar to my lab and max out at a DGX node rather than being the bulk of users. PyTorch has gotten a lot better on TPUs this year, I don’t believe there’s much of a performance hit now. Jax and TF (I don’t use the latter anymore) of course work. I never used Gaudi2 but it apparently works. All of this to say, is it possible that Intel gets their fabs working and strategically partners with their long-time partner MS and OpenAI to extend Triton to Gaudi3 or 4 and be a threat to Nvidia within 2-3 years? Absolutely. Is it similarly possible that Google increases development on Jax and TPUv5? Sure. Neither of these possibilities, regardless if you think they’re probable or improbable, would need a decade to catch up to Nvidia.
    - danielmarkbruce 3y ago
      You may well be right. It's going to be an interesting few years.

haldujai 3y ago

> You seem to be moving the goalposts. First it was using in production, now it's building foundation models.. I'm not moving the goal posts but maybe I wasn't clear. When we talk about corporations adopting fine-tuning and inference that's not especially relevant to H100 sales which is the main cash-cow (~10B of revenue at 80-90% margins) and Nvidia's massive market cap growth. What is relevant is corporations like Inflection AI building 22k H100 clusters. I work in academia so PyTorch is more common but are people in industry fine-tuning LLMs actually working directly with CUDA that much for this to be a big moat?

danielmarkbruce 3y ago
Companies are using both A100 and H100 for inference. The datacenter numbers include both, and I don't believe they break it down further. And no, by and large, large enterprises are not building out large clusters - but they are much of the demand for all those cloud provider build outs. No, no one is using CUDA directly. But if you are a vendor with a new equivalent, it's no small feat integrating it with a framework like PyTorch. There is a lot of work required by various parties.
- haldujai 3y ago
  It would be interesting to see what the breakdown is on H100/A100 users. I would expect most inference users are similar to my lab and max out at a DGX node rather than being the bulk of users. PyTorch has gotten a lot better on TPUs this year, I don’t believe there’s much of a performance hit now. Jax and TF (I don’t use the latter anymore) of course work. I never used Gaudi2 but it apparently works. All of this to say, is it possible that Intel gets their fabs working and strategically partners with their long-time partner MS and OpenAI to extend Triton to Gaudi3 or 4 and be a threat to Nvidia within 2-3 years? Absolutely. Is it similarly possible that Google increases development on Jax and TPUv5? Sure. Neither of these possibilities, regardless if you think they’re probable or improbable, would need a decade to catch up to Nvidia.
  - danielmarkbruce 3y ago
    You may well be right. It's going to be an interesting few years.