4 ms·

> Engineers at hyperscalers are struggling through all the bugs too [citation needed]

by sangnoir 2y ago

almostgotcaught 2y ago
Lolol 100% accurate. Go trawl through PRs to Triton by FB people to the AMD portion of the codebase.
Twirrim 2y ago
Sorry, but NDAs mean I can't say any more.
mike_hearn 2y ago
It's in the article. Meta don't use AMD for training and write their own kernels for inference. You can't train with AMD, full stop, because their software stack is so buggy.
- sangnoir 2y ago
  > It's in the article The same article also states that AMD provided custom bug-fixes written by Principle Engineers to address bugs in a benchmark - this is software that will only become part of the public release in 2 quarters. I ask again, do you think AMD will not expedite non-public bug-fixes for hyperscalers? > You can't train with AMD, full stop, because their software stack is so buggy. Point 7 from the article: >> The MI300X has a lower total cost of ownership (TCO) compared to the H100/H200, but training performance per TCO is worse on the MI300X on public stable releases of AMD software. This changes if one uses custom development builds of AMD software. I have an inkling that Meta does not obtain MI300 drivers from https://download.amd.com https://download.amd.com
  - mike_hearn 2y ago
    AI labs don't want to train models using a stack build some guy hacked up on his desktop last night that's been through no proper QA process. The cost of a training run that fails or results in a garbage model due to numerical errors are huge. Which is why, as they say clearly, nobody is training models on AMD. Only inference, at most. I'm not sure why you keep claiming they are training using private drivers. They clearly aren't.
    - sangnoir 2y ago
      Now I i see how where talking past each other. I 100% agree that none of if the hyperscalers are currently (publicly) training on AMD silicon. I disagree with forward-looking statements like it "can't" happen, because I can guarantee you several of them are actively working on making it possible to train on AMD chips - that's just too juicy a target for a bonus packets all the way up to directors: "Our team saved the org $x0 million in TCO vy enabling training on MI300/MI400X in our new clusters"
      - mike_hearn 2y ago
        Oh, I see. "Can't" means in the present tense in my previous sentence, it wasn't meant to be a definitive statement about the entire future.

> Engineers at hyperscalers are struggling through all the bugs too [citation needed]

almostgotcaught 2y ago

Lolol 100% accurate. Go trawl through PRs to Triton by FB people to the AMD portion of the codebase.

Twirrim 2y ago

Sorry, but NDAs mean I can't say any more.

mike_hearn 2y ago

It's in the article. Meta don't use AMD for training and write their own kernels for inference. You can't train with AMD, full stop, because their software stack is so buggy.

sangnoir 2y ago
> It's in the article The same article also states that AMD provided custom bug-fixes written by Principle Engineers to address bugs in a benchmark - this is software that will only become part of the public release in 2 quarters. I ask again, do you think AMD will not expedite non-public bug-fixes for hyperscalers? > You can't train with AMD, full stop, because their software stack is so buggy. Point 7 from the article: >> The MI300X has a lower total cost of ownership (TCO) compared to the H100/H200, but training performance per TCO is worse on the MI300X on public stable releases of AMD software. This changes if one uses custom development builds of AMD software. I have an inkling that Meta does not obtain MI300 drivers from https://download.amd.com https://download.amd.com
- mike_hearn 2y ago
  AI labs don't want to train models using a stack build some guy hacked up on his desktop last night that's been through no proper QA process. The cost of a training run that fails or results in a garbage model due to numerical errors are huge. Which is why, as they say clearly, nobody is training models on AMD. Only inference, at most. I'm not sure why you keep claiming they are training using private drivers. They clearly aren't.
  - sangnoir 2y ago
    Now I i see how where talking past each other. I 100% agree that none of if the hyperscalers are currently (publicly) training on AMD silicon. I disagree with forward-looking statements like it "can't" happen, because I can guarantee you several of them are actively working on making it possible to train on AMD chips - that's just too juicy a target for a bonus packets all the way up to directors: "Our team saved the org $x0 million in TCO vy enabling training on MI300/MI400X in our new clusters"
    - mike_hearn 2y ago
      Oh, I see. "Can't" means in the present tense in my previous sentence, it wasn't meant to be a definitive statement about the entire future.