6 ms·

Many labs use TPUs, but not exclusively. Most labs need more compute than they can get, and if there's TPU capacity, they'll adapt their systems to be able to r

Many labs use TPUs, but not exclusively. Most labs need more compute than they can get, and if there's TPU capacity, they'll adapt their systems to be able to run partially on TPUs.

gpt5 6mo ago
Why is AMD not more popular then if labs are so flexibly with giving away CUDA?
- mattnewton 6mo ago
  people are trying, especially for inference. For training, it’s just too high risk to tank your training I think. TPUs are at least dogfooded by Google deepmind, no team AFAIK has gotten the AMD stack to train well.
  - coder-3 6mo ago
    Interesting. Why? My current mental model is that AMD chips are just a bit behind, so, less efficient, but no biggie. Do labs even use CUDA?
    - 0-_-0 6mo ago
      What I hear is that getting your network to work on AMD is a huge pain.
      - dnadler 6mo ago
        Yeah, historically it’s been software that’s limited AMD here. Not surprised to hear that may still be the issue. NVidia’s biggest edge was really CUDA.
        oomuinio 6mo ago
        [dead]
        otabdeveloper4 6mo ago
        CUDA is a complete and utter piece of shit software. It's just that it is a tiny bit less of a shitshow than the alternatives.
    - uberduper 6mo ago
      amd gpus compete but they lack the interconnect. NVLink performance is a huge deal for training.
    - nl 6mo ago
      This is somewhat out of date (Dec 2024), but gives you some idea of how far behind AMD was then: https://newsletter.semianalysis.com/p/mi300x-vs-h100-vs-h200-benchmark-part-1-training https://newsletter.semianalysis.com/p/mi300x-vs-h100-vs-h200... Pull quotes: AMD’s software experience is riddled with bugs rendering out of the box training with AMD is impossible. We were hopeful that AMD could emerge as a strong competitor to NVIDIA in training workloads, but, as of today, this is unfortunately not the case. The CUDA moat has yet to be crossed by AMD due to AMD’s weaker-than-expected software Quality Assurance (QA) culture and its challenging out of the box experience. [snip] > The only reason we have been able to get AMD performance within 75% of H100/H200 performance is because we have been supported by multiple teams at AMD in fixing numerous AMD software bugs. To get AMD to a usable state with somewhat reasonable performance, a giant ~60 command Dockerfile that builds dependencies from source, hand crafted by an AMD principal engineer, was specifically provided for us [snip] > AMD hipBLASLt/rocBLAS’s heuristic model picks the wrong algorithm for most shapes out of the box, which is why so much time-consuming tuning is required by the end user. etc etc. The whole thing is worth reading. I'm sure it has (and will continue to) improved since then. I hear good things about the Lemonade team (although I think that is mostly inference?) But the NVidia stack has improved too.
      - _vertigo 6mo ago
        That’s insane. There should be a big team of people at AMD whose whole job is just to dogfood their stuff for training like this. Speaking of which, Amazon is in the same boat, I’m constantly surprised that Amazon is not treating improving Inferentia/Trainium software as an uber-priority. (I work at Amazon)
        chii 6mo ago
        > There should be a big team of people at AMD whose whole job is just to dogfood their stuff if they had this management attitude, they wouldn't have been so far behind so as to need this action in the first place!
        nl 6mo ago
    - electroglyph 6mo ago
      i'm doing inference on a free mi300x instance from AMD right now. not sure if the software stack is just old or what, but here's what i've observed: stuck on an old version of vllm pre-Transformers 5 support. it lacks MoE support for qwen3 models. oss-120b is faaaar slower than it should be. int8 quantization seems like it's almost supported, but not quite. speeds drop to a fraction of full precision speed and the server seems like it intermittently hangs. int4 quantization not supported. fp8 quantization not supported. again, maybe AMD is just being lazy with what they've provided, but it's not a great look. right now the fastest smart model i can run is full precision qwen3-32b. with 120 parallel requests (short context) i'm getting PP @ 4500 tokens/sec and TG @ 1300 tokens/sec
    - bean469 6mo ago
      > Do labs even use CUDA? From the papers I've read and the labs that I have worked in personally, I would say that most scientists developing Deep learning solutions use CUDA for GPU acceleration
    - f6v 6mo ago
      I don’t know what’s a chicken and what’s an egg here. But ROCm support is often missing or experimental even in very basic foundational libraries. They need someone else to double down on using their chips and just break the software support out of the limbo.
    - mlmonkey 6mo ago
      This is what I've heard on the "street". Building a CUDA-compatible stack for AMD's hardware requires highly-paid SWEs. It's a very niche field, and talent is hard to come by. But AMD does not want to pay these specialized SWEs the market rate. Their existing SWEs would be up in arms saying, basically, "what are we, chopped liver??", or so the thinking goes. So AMD is stuck with a shitty software stack which cannot compete with CUDA. If I were making such decisions, I would just cull the number of existing SWEs down by 50%, and double the pay for remaining ones. And then go out and hire some top talent to build a good software stack.
      - deleted 6mo ago
        [deleted]
      - square_usual 6mo ago
        > highly-laid SWEs Freudian slip?
        mlmonkey 6mo ago
        Ha! You caught it before I did; and I caught it right away.
        fuck_google 6mo ago
        [dead]
      - hephaes7us 6mo ago
        [dead]
- zombiwoof 6mo ago
  [dead]
yjadsfgasdf 6mo ago
[dead]
zobzu 6mo ago
even google doesnt only use TPUs.
- danpalmer 6mo ago
  Google is in a different position to others in that they're the only frontier lab with a cloud infra business. It obviously makes sense to sell GPUs on cloud infra as people want to rent them. In that respect Google buys a ton of GPUs to rent out. What's unclear to me is how much Google uses GPUs for their own stuff. Yes Gemini runs on GPUs now, so that Google can sell Gemini on-prem boxes (recent release announced last week), but is any training or inference for Gemini really happening on GPUs? This is unclear to me. I'd have guessed not given that I thought TPUs were much cheaper to operate, but maybe I'm wrong. Caveat, I work at Google, but not on anything to do with this. I'm only going on what's in the press for this stuff.
  - bartwr 6mo ago
    I have most likely outdated info, I left Google Research 4y ago. Back then, available TPU instances were plenty and GPU scarce. Nobody wanted to mess with an immature crashing compiler and very steep performance cliffs (performance was excellent only if you stayed within the guardrails, and being outside was supported and not even resulting in a warning - as it was so common in code). But I believe most of it has changed for the better for TPUs.
  - johndough 6mo ago
    > Gemini on-prem boxes (recent release announced last week) Do you have any more information on this? I only found this article about it: https://venturebeat.com/technology/googles-gemini-can-now-run-on-a-single-air-gapped-server-and-vanish-when-you-pull-the-plug https://venturebeat.com/technology/googles-gemini-can-now-ru... It mentions that Gemini can run on eight NVIDIA GPUs, but not which GPU and which Gemini model. Either way, this puts an upper bound of 288 * 8 = 2304 GB on the size of the Gemini model, which as far as I know has been a secret until now.
    - danpalmer 6mo ago

Many labs use TPUs, but not exclusively. Most labs need more compute than they can get, and if there's TPU capacity, they'll adapt their systems to be able to run partially on TPUs.

gpt5 6mo ago

Why is AMD not more popular then if labs are so flexibly with giving away CUDA?

mattnewton 6mo ago
people are trying, especially for inference. For training, it’s just too high risk to tank your training I think. TPUs are at least dogfooded by Google deepmind, no team AFAIK has gotten the AMD stack to train well.
- coder-3 6mo ago
  Interesting. Why? My current mental model is that AMD chips are just a bit behind, so, less efficient, but no biggie. Do labs even use CUDA?
  - 0-_-0 6mo ago
    What I hear is that getting your network to work on AMD is a huge pain.
    - dnadler 6mo ago
      Yeah, historically it’s been software that’s limited AMD here. Not surprised to hear that may still be the issue. NVidia’s biggest edge was really CUDA.
      - oomuinio 6mo ago
        [dead]
      - otabdeveloper4 6mo ago
        CUDA is a complete and utter piece of shit software. It's just that it is a tiny bit less of a shitshow than the alternatives.
  - uberduper 6mo ago
    amd gpus compete but they lack the interconnect. NVLink performance is a huge deal for training.
  - nl 6mo ago
    This is somewhat out of date (Dec 2024), but gives you some idea of how far behind AMD was then: https://newsletter.semianalysis.com/p/mi300x-vs-h100-vs-h200-benchmark-part-1-training https://newsletter.semianalysis.com/p/mi300x-vs-h100-vs-h200... Pull quotes: AMD’s software experience is riddled with bugs rendering out of the box training with AMD is impossible. We were hopeful that AMD could emerge as a strong competitor to NVIDIA in training workloads, but, as of today, this is unfortunately not the case. The CUDA moat has yet to be crossed by AMD due to AMD’s weaker-than-expected software Quality Assurance (QA) culture and its challenging out of the box experience. [snip] > The only reason we have been able to get AMD performance within 75% of H100/H200 performance is because we have been supported by multiple teams at AMD in fixing numerous AMD software bugs. To get AMD to a usable state with somewhat reasonable performance, a giant ~60 command Dockerfile that builds dependencies from source, hand crafted by an AMD principal engineer, was specifically provided for us [snip] > AMD hipBLASLt/rocBLAS’s heuristic model picks the wrong algorithm for most shapes out of the box, which is why so much time-consuming tuning is required by the end user. etc etc. The whole thing is worth reading. I'm sure it has (and will continue to) improved since then. I hear good things about the Lemonade team (although I think that is mostly inference?) But the NVidia stack has improved too.
    - _vertigo 6mo ago
      That’s insane. There should be a big team of people at AMD whose whole job is just to dogfood their stuff for training like this. Speaking of which, Amazon is in the same boat, I’m constantly surprised that Amazon is not treating improving Inferentia/Trainium software as an uber-priority. (I work at Amazon)
      - chii 6mo ago
        > There should be a big team of people at AMD whose whole job is just to dogfood their stuff if they had this management attitude, they wouldn't have been so far behind so as to need this action in the first place!
        nl 6mo ago
  - electroglyph 6mo ago
    i'm doing inference on a free mi300x instance from AMD right now. not sure if the software stack is just old or what, but here's what i've observed: stuck on an old version of vllm pre-Transformers 5 support. it lacks MoE support for qwen3 models. oss-120b is faaaar slower than it should be. int8 quantization seems like it's almost supported, but not quite. speeds drop to a fraction of full precision speed and the server seems like it intermittently hangs. int4 quantization not supported. fp8 quantization not supported. again, maybe AMD is just being lazy with what they've provided, but it's not a great look. right now the fastest smart model i can run is full precision qwen3-32b. with 120 parallel requests (short context) i'm getting PP @ 4500 tokens/sec and TG @ 1300 tokens/sec
  - bean469 6mo ago
    > Do labs even use CUDA? From the papers I've read and the labs that I have worked in personally, I would say that most scientists developing Deep learning solutions use CUDA for GPU acceleration
  - f6v 6mo ago
    I don’t know what’s a chicken and what’s an egg here. But ROCm support is often missing or experimental even in very basic foundational libraries. They need someone else to double down on using their chips and just break the software support out of the limbo.
  - mlmonkey 6mo ago
    This is what I've heard on the "street". Building a CUDA-compatible stack for AMD's hardware requires highly-paid SWEs. It's a very niche field, and talent is hard to come by. But AMD does not want to pay these specialized SWEs the market rate. Their existing SWEs would be up in arms saying, basically, "what are we, chopped liver??", or so the thinking goes. So AMD is stuck with a shitty software stack which cannot compete with CUDA. If I were making such decisions, I would just cull the number of existing SWEs down by 50%, and double the pay for remaining ones. And then go out and hire some top talent to build a good software stack.
    - deleted 6mo ago
      [deleted]
    - square_usual 6mo ago
      > highly-laid SWEs Freudian slip?
      - mlmonkey 6mo ago
        Ha! You caught it before I did; and I caught it right away.
      - fuck_google 6mo ago
        [dead]
    - hephaes7us 6mo ago
      [dead]
zombiwoof 6mo ago
[dead]

yjadsfgasdf 6mo ago

[dead]

zobzu 6mo ago

even google doesnt only use TPUs.

danpalmer 6mo ago
Google is in a different position to others in that they're the only frontier lab with a cloud infra business. It obviously makes sense to sell GPUs on cloud infra as people want to rent them. In that respect Google buys a ton of GPUs to rent out. What's unclear to me is how much Google uses GPUs for their own stuff. Yes Gemini runs on GPUs now, so that Google can sell Gemini on-prem boxes (recent release announced last week), but is any training or inference for Gemini really happening on GPUs? This is unclear to me. I'd have guessed not given that I thought TPUs were much cheaper to operate, but maybe I'm wrong. Caveat, I work at Google, but not on anything to do with this. I'm only going on what's in the press for this stuff.
- bartwr 6mo ago
  I have most likely outdated info, I left Google Research 4y ago. Back then, available TPU instances were plenty and GPU scarce. Nobody wanted to mess with an immature crashing compiler and very steep performance cliffs (performance was excellent only if you stayed within the guardrails, and being outside was supported and not even resulting in a warning - as it was so common in code). But I believe most of it has changed for the better for TPUs.
- johndough 6mo ago
  > Gemini on-prem boxes (recent release announced last week) Do you have any more information on this? I only found this article about it: https://venturebeat.com/technology/googles-gemini-can-now-run-on-a-single-air-gapped-server-and-vanish-when-you-pull-the-plug https://venturebeat.com/technology/googles-gemini-can-now-ru... It mentions that Gemini can run on eight NVIDIA GPUs, but not which GPU and which Gemini model. Either way, this puts an upper bound of 288 * 8 = 2304 GB on the size of the Gemini model, which as far as I know has been a secret until now.
  - danpalmer 6mo ago