6 ms·

Does it not work for them? Where can I learn why?

by Hugsun 3y ago

tucnak 3y ago
Just go have a look around Github issues in their ROCm repositories on Github. A few months back the top excuse re: AMD was that we're not supposed to use their "consumer" cards, however the datacenter stuff is kosher. Well, guess what, we have purchased their datacenter card, MI50, and it's similarly screwed. Too many bugs in the kernel, kernel crashes, hangs, and the ROCm code is buggy / incomplete. When it works, it works for a short period of time, and yes HBM memory is kind of nice, but the whole thing is not worth it. Some say MI210 and MI300 are better, but it's just wishful thinking as all the bugs are in the software, kernel driver, and firmware. I have spent too many hours troubleshooting entry-level datacenter-grade Instinct cards with no recourse from AMD whatsoever to pay 10+ thousands for MI210 a couple-year old underpowered hardware, and MI300 is just unavailable. Not even from cloud providers which should be telling enough.
- latchkey 3y ago
  [flagged]
  - buildbot 3y ago
    Probably not, I have had the same experience with a 780m and a Mi60...
  - Scaevolus 3y ago
    Effectively everyone that has attempted to use AMD hardware for ML comes away with these opinions, the main difference is how angrily they express it.
    - sangnoir 3y ago
      So who's buying all the MI300s? Groq seems to be fine with AMD.
      - tucnak 3y ago
        Groq doesn't use AMD afaik, they had designed hardware of their own, which is actually 1000s of SRAM chips in a trench-coat.
        latchkey 3y ago
        Speaking of SRAM, I found this relevant comment insightful: https://news.ycombinator.com/item?id=39966620 https://news.ycombinator.com/item?id=39966620
      - latchkey 3y ago
        > So who's buying all the MI300s? We are. Just closed additional funding and getting quotes now.
    - latchkey 3y ago
      We are the 4th non-hyperscaler business on the planet to even get access to MI300x and we just got it in early March. From what I understand, hyperscalers have had fantastic uptake of this hardware. I find it hard to believe "everyone" comes away with these opinions. https://www.evp.cloud/post/diving-deeper-insights-from-our-llm-inference-testing https://www.evp.cloud/post/diving-deeper-insights-from-our-l...
- Workaccount2 3y ago
  It's seriously impressive how well AMD has been able to maintain their incredible software deficiency for over a decade now.
  - alexey-salmin 3y ago
    They deeply care about the tradition of ATI kernel modules from 2004
    - sumtechguy 3y ago
      more like 1998 :)
  - amirhirsch 3y ago
    Buying Xilinx helped a lot here.
    - fpgamlirfanboy 3y ago
      it's so true it hurts
      - tucnak 3y ago
        Hey man have seen you around here, very knowledgeable, thanks for your input! What's your take on projects like https://github.com/corundum/corundum https://github.com/corundum/corundum I'm trying to get better at FPGA design, perhaps learn PCIe and some such but Vivado is intimidating (as opposed to Yosys/nextpnr which you seem to hate) should I just get involved with a project like this to acclimatise somewhat?
        fpgamlirfanboy 3y ago
        > Vivado is intimidating (as opposed to Yosys/nextpnr which you seem to hate) i never said i hated yosys/nextpnr? i said somewhere that yosys makes the uber strange decision to use C++ as effectively a scripting language ie gluing and scheduling "passes" together - like they seemed to make the firm decision to diverge from tcl but diverged into absurd territory. i wish yosys were great because it's open source and then i could solve my own problems as they occurred. but it's not great and i doubt it ever will be because building logic synthesis, techmapping, timing analysis, place and route, etc. is just too many extremely hard problems for OSS. all the vendor tools suck. it's just a fact that both big fpga manufacturers have completely shit software devs working on those tools. the only tools that i've heard are decent are the very expensive suites from cadence/siemenns/synopsis but i have yet to be in a place that has licenses (neither school nor day job - at least not in my team). and mind you, you will still need to feed the RTL or netlist or whatever that those tools generate into vivado (so you're still fucked). so i don't have advice for you on RTL - i moved one level up (ISA, compilers, etc.) primarily because i could not effectively learn by myself i.e., without going to "apprentice" under someone that just has enough experience to navigate around the potholes (because fundamentally if that's what it takes to learn then you're basically working on ineluctably entrenched tech).
- jmward01 3y ago
  Yeah, this has stopped me from trying anything with them. They need to lead with their consumer cards so that developers can test/build/evaluate/gain trust locally and then their enterprise offerings need to 100% guarantee that the stuff developers worked on will work in the data center. I keep hoping to see this but every time I look it isn't there. There is way more support for apple silicon out there than ROCm and that has no path to enterprise. AMD is missing the boat.
  - latchkey 3y ago
    You are right, AMD should do more with consumer cards, but I understand why they aren't today. It is a big ship, they've really only started changing course as of last Oct/Nov, before the release of MI300x in Dec. If you have limited resources and a whole culture to change, you have to give them time to fix that. That said, if you're on the inside, like I am, and you talk to people at AMD (just got off two separate back to back calls with them), rest assured, they are dedicated to making this stuff work. Part of that is to build a developer flywheel by making their top end hardware available to end users. That's where my company Hot Aisle comes into play. Something that wasn't available before outside of the HPC markets, is now going to be made available.
    - tucnak 3y ago
      [flagged]
      - latchkey 3y ago
        https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
    - jmward01 3y ago
      I look forward to seeing it. NVIDIA needs real competition for their own benefit if not the market as a whole. I want a richer ecosystem where Intel, AMD, NVIDIA and other players all join in with the winner being the consumer. From a selfish point of view I also want to do more home experimentation. LLMs are so new that you can make breakthroughs without a huge team but it really helps to have hardware to make it easier to play with ideas. Consumer card memory limitations are hurting that right now.
- JonChesterfield 3y ago
  We absolutely hammered the MI50 in internal testing for ages. Was solid as far as I can tell. Rocm is sensitive to matching kernel version to driver version to userspace version. Staying very much on the kernel version from a official release and using the corresponding driver is drastically more robust than optimistically mixing different components. In particular, rocm is released and tested as one large blob, and running that large blob on a slightly different kernel version can go very badly. Mixing things from GitHub with things from your package manager is also optimistic. Imagine it as huge ball of code where cross version compatibility of pieces is totally untested.
  - tucnak 3y ago
    I would run simple llama.cpp batch jobs for 10 minutes when it would suddenly fail, and require a restart. Random VM_L2_PROTECTION_FAULT in dmesg, something having to do with doorbells. I did report this, never heard back from them.
    - latchkey 3y ago
      Would you like to share the model of GPU and versions of various software used? George has a nice explanation of doorbells: https://youtu.be/AqPIOtUkxNo?feature=shared&t=968 https://youtu.be/AqPIOtUkxNo?feature=shared&t=968
    - FeepingCreature 3y ago
      Same here with SD on 7900XTX. Most of the time for me it's sufficient to reset the card with rocm-smi --gpureset -d 0.
      - michaelt 3y ago
        Only "most of the time" ? :( You'd hope at $15,000+ per unit, you wouldn't have to reset it at all...
        JonChesterfield 3y ago
- cavisne 3y ago
  Yeah, I think AMD will really struggle with the cloud providers. Even Nvidia GPU's are tricky to sandbox, and it sounds like the AMD cards are really easy for the tenant to break (or at least force a restart of the underlying host). AWS does have a Gaudi instance which is interesting, but overall I don't see why Azure, AWS & Google would deploy AMD or Intel GPU's at scale vs their own chips. They need some competitor to Nvidia to help negotiate, but if its going to be a painful software support story suited to only a few enterprise customers, why not do it with your own chip?

Does it not work for them? Where can I learn why?

by Hugsun 3y ago

Does it not work for them? Where can I learn why?

Just go have a look around Github issues in their ROCm repositories on Github. A few months back the top excuse re: AMD was that we're not supposed to use their "consumer" cards, however the datacenter stuff is kosher. Well, guess what, we have purchased their datacenter card, MI50, and it's similarly screwed. Too many bugs in the kernel, kernel crashes, hangs, and the ROCm code is buggy / incomplete. When it works, it works for a short period of time, and yes HBM memory is kind of nice, but the whole thing is not worth it. Some say MI210 and MI300 are better, but it's just wishful thinking as all the bugs are in the software, kernel driver, and firmware. I have spent too many hours troubleshooting entry-level datacenter-grade Instinct cards with no recourse from AMD whatsoever to pay 10+ thousands for MI210 a couple-year old underpowered hardware, and MI300 is just unavailable. Not even from cloud providers which should be telling enough.

latchkey 3y ago
[flagged]
- buildbot 3y ago
  Probably not, I have had the same experience with a 780m and a Mi60...
- Scaevolus 3y ago
  Effectively everyone that has attempted to use AMD hardware for ML comes away with these opinions, the main difference is how angrily they express it.
  - sangnoir 3y ago
    So who's buying all the MI300s? Groq seems to be fine with AMD.
    - tucnak 3y ago
      Groq doesn't use AMD afaik, they had designed hardware of their own, which is actually 1000s of SRAM chips in a trench-coat.
      - latchkey 3y ago
        Speaking of SRAM, I found this relevant comment insightful: https://news.ycombinator.com/item?id=39966620 https://news.ycombinator.com/item?id=39966620
    - latchkey 3y ago
      > So who's buying all the MI300s? We are. Just closed additional funding and getting quotes now.
  - latchkey 3y ago
    We are the 4th non-hyperscaler business on the planet to even get access to MI300x and we just got it in early March. From what I understand, hyperscalers have had fantastic uptake of this hardware. I find it hard to believe "everyone" comes away with these opinions. https://www.evp.cloud/post/diving-deeper-insights-from-our-llm-inference-testing https://www.evp.cloud/post/diving-deeper-insights-from-our-l...
Workaccount2 3y ago
It's seriously impressive how well AMD has been able to maintain their incredible software deficiency for over a decade now.
- alexey-salmin 3y ago
  They deeply care about the tradition of ATI kernel modules from 2004
  - sumtechguy 3y ago
    more like 1998 :)
- amirhirsch 3y ago
  Buying Xilinx helped a lot here.
  - fpgamlirfanboy 3y ago
    it's so true it hurts
    - tucnak 3y ago
      Hey man have seen you around here, very knowledgeable, thanks for your input! What's your take on projects like https://github.com/corundum/corundum https://github.com/corundum/corundum I'm trying to get better at FPGA design, perhaps learn PCIe and some such but Vivado is intimidating (as opposed to Yosys/nextpnr which you seem to hate) should I just get involved with a project like this to acclimatise somewhat?
      - fpgamlirfanboy 3y ago
        > Vivado is intimidating (as opposed to Yosys/nextpnr which you seem to hate) i never said i hated yosys/nextpnr? i said somewhere that yosys makes the uber strange decision to use C++ as effectively a scripting language ie gluing and scheduling "passes" together - like they seemed to make the firm decision to diverge from tcl but diverged into absurd territory. i wish yosys were great because it's open source and then i could solve my own problems as they occurred. but it's not great and i doubt it ever will be because building logic synthesis, techmapping, timing analysis, place and route, etc. is just too many extremely hard problems for OSS. all the vendor tools suck. it's just a fact that both big fpga manufacturers have completely shit software devs working on those tools. the only tools that i've heard are decent are the very expensive suites from cadence/siemenns/synopsis but i have yet to be in a place that has licenses (neither school nor day job - at least not in my team). and mind you, you will still need to feed the RTL or netlist or whatever that those tools generate into vivado (so you're still fucked). so i don't have advice for you on RTL - i moved one level up (ISA, compilers, etc.) primarily because i could not effectively learn by myself i.e., without going to "apprentice" under someone that just has enough experience to navigate around the potholes (because fundamentally if that's what it takes to learn then you're basically working on ineluctably entrenched tech).
jmward01 3y ago
Yeah, this has stopped me from trying anything with them. They need to lead with their consumer cards so that developers can test/build/evaluate/gain trust locally and then their enterprise offerings need to 100% guarantee that the stuff developers worked on will work in the data center. I keep hoping to see this but every time I look it isn't there. There is way more support for apple silicon out there than ROCm and that has no path to enterprise. AMD is missing the boat.
- latchkey 3y ago
  You are right, AMD should do more with consumer cards, but I understand why they aren't today. It is a big ship, they've really only started changing course as of last Oct/Nov, before the release of MI300x in Dec. If you have limited resources and a whole culture to change, you have to give them time to fix that. That said, if you're on the inside, like I am, and you talk to people at AMD (just got off two separate back to back calls with them), rest assured, they are dedicated to making this stuff work. Part of that is to build a developer flywheel by making their top end hardware available to end users. That's where my company Hot Aisle comes into play. Something that wasn't available before outside of the HPC markets, is now going to be made available.
  - tucnak 3y ago
    [flagged]
    - latchkey 3y ago
      https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
  - jmward01 3y ago
    I look forward to seeing it. NVIDIA needs real competition for their own benefit if not the market as a whole. I want a richer ecosystem where Intel, AMD, NVIDIA and other players all join in with the winner being the consumer. From a selfish point of view I also want to do more home experimentation. LLMs are so new that you can make breakthroughs without a huge team but it really helps to have hardware to make it easier to play with ideas. Consumer card memory limitations are hurting that right now.
JonChesterfield 3y ago
We absolutely hammered the MI50 in internal testing for ages. Was solid as far as I can tell. Rocm is sensitive to matching kernel version to driver version to userspace version. Staying very much on the kernel version from a official release and using the corresponding driver is drastically more robust than optimistically mixing different components. In particular, rocm is released and tested as one large blob, and running that large blob on a slightly different kernel version can go very badly. Mixing things from GitHub with things from your package manager is also optimistic. Imagine it as huge ball of code where cross version compatibility of pieces is totally untested.
- tucnak 3y ago
  I would run simple llama.cpp batch jobs for 10 minutes when it would suddenly fail, and require a restart. Random VM_L2_PROTECTION_FAULT in dmesg, something having to do with doorbells. I did report this, never heard back from them.
  - latchkey 3y ago
    Would you like to share the model of GPU and versions of various software used? George has a nice explanation of doorbells: https://youtu.be/AqPIOtUkxNo?feature=shared&t=968 https://youtu.be/AqPIOtUkxNo?feature=shared&t=968
  - FeepingCreature 3y ago
    Same here with SD on 7900XTX. Most of the time for me it's sufficient to reset the card with rocm-smi --gpureset -d 0.
    - michaelt 3y ago
      Only "most of the time" ? :( You'd hope at $15,000+ per unit, you wouldn't have to reset it at all...
      - JonChesterfield 3y ago
cavisne 3y ago
Yeah, I think AMD will really struggle with the cloud providers. Even Nvidia GPU's are tricky to sandbox, and it sounds like the AMD cards are really easy for the tenant to break (or at least force a restart of the underlying host). AWS does have a Gaudi instance which is interesting, but overall I don't see why Azure, AWS & Google would deploy AMD or Intel GPU's at scale vs their own chips. They need some competitor to Nvidia to help negotiate, but if its going to be a painful software support story suited to only a few enterprise customers, why not do it with your own chip?