5 ms·

AMD Instinct MI300X Accelerators

36 points by bx376 3y ago

frognumber 3y ago
I stopped supporting AMD when I bought an AMD card, and they discontinued support a few months later. If AMD sends me an MI300X for free, I'll make sure my software is compatible. Otherwise, if you want to run my stuff, you need Nvidia (or, hopefully soon, Intel). (Note: I know they never will. This story, multiplied by the number of people building stuff now, is why I'm bearish on AMD and bullish on competitors. This would be a very nice card -- 192GB is sexy -- but I just get headaches thinking about compatibility, software, and drivers issues. To quote Steve Balmer, what AMD is missing is: "Developers, developers, developers." That's not a small hole to dig out of.)
- vlakreeh 3y ago
  What card could you have bought that lost support months later? I can't think of any new card you could have bought that'd lose support that quickly as far back as 2016
  - KeplerBoy 3y ago
    Not GP, but they dropped support for SKUs which were still sold at that point. The MI50 would be a fun accelerator to play with and easily worth 2000$ if it were made by Nvidia. Since they are made by AMD and not supported, you can grab them for 200$. https://github.com/ROCm/ROCm/issues/2308 https://github.com/ROCm/ROCm/issues/2308
    - simfree 3y ago
      Support still exists, but the card is in maintenance mode. This doesn't mean that your models that work on an MI50 today are suddenly going to stop working, or the models ran on my MI25 will cease suddenly.
      - frognumber 3y ago
        That's only true if you don't mind sharing your systems with strangers in Russia, Asia, and Africa. "Bug fixes / critical security patches will continue to be supported for the gfx906 GPUs till Q2 2024" Worse, a lot of tooling will break before then. If you need a security patch for anything built on top of ROCm or CUDA, you often can't have an ancient version. That's not to mention that I'd like to do new development and prototyping. I don't do business with AMD for the same reason I don't do business with Google. They're not a reliable partner.
        nixgeek 3y ago
        That the only bad actors are in Russia, Asia and Africa is becoming quite the trope. Some very spectacular compromises by Lapsus$ aren’t originating from any of the above, but instead from the West, specifically the UK: https://www.bbc.com/news/technology-67663128 https://www.bbc.com/news/technology-67663128
        frognumber 3y ago
        I think you're missing the point. Bad actors are equally distributed around the world. The difference, however, is that, in this case, as the article points out, the bad actor is in custody. My local law enforcement, lawyer, and judicial system are able to help out with the bad actors here and keep the problem somewhat contained. For a bad actor in Russia or India, you quite literally have no recourse. There are literally Youtube videos (Mark Rober did a special recently) showing out-of-jurisdiction scammer and hacker shops occupying entire office buildings, and there is almost nothing we can do about it. I'm not even going to go on a diatribe about corruption and the quality of law enforcement, since one can have different opinions. What I will point out is that US+EU have extensive treaties which allow my local law enforcement agency to cooperate closely with law enforcement agencies between US, UK, and EU. For a bad actor in Iran or North Korea? You're not even getting diplomatic contact. Footnote: I mention India since they're a democracy and by all standards, try to act responsibly on the global stage. They just don't happen to be in-network for various US/EU-centric international conventions.
        3y ago
  - frognumber 3y ago
    https://github.com/ROCm/ROCm/issues/1353 https://github.com/ROCm/ROCm/issues/1353 Bought in 2020. Stopped working in 2020. Not the latest, but in-production, advertised ROCm-capable, and what I could find during the Great GPU Shortage of 2020.
    - justinclift 3y ago
      Wow, not even an industry-standard-for-many-decades deprecation notice up front, to give people a heads up before hand. That's pretty fucked. :(
      - frognumber 3y ago
        It's even worse. Order of operations: 1. It just silently stopped working, started crashing, and I had no idea what was going on or why. Just sort of intermittent f-age where some older versions would work better, and newer ones would work worse. I never got it reliably working. In most cases, at some point, the system would become unresponsive, and then crash hard. 2. AMD removed it from the supported list (with no notice). 3. The github ticket above was filed, which explained what was going on. Lots of time wasted debugging. This interacted with a half-dozen other AMD bugs and issues (such as learning the GPU only worked for compute headless; I needed to drive my monitor with a different card). Human time is more expensive than equipment, so the total cost of this stuff was astronomical. Yes, notice is industry-standard, but at the very least, when support broke / was later removed, the driver could try warning me "We no longer support this card, do you REALLY want to proceed?" rather than letting me know by hard-crashing my system.
        imtringued 3y ago
        People don't seem to understand how ROCm fails. Some inaccessible list somewhere buried deep down or a GitHub issues says your card is dropped. Apparently the average user is supposed to spend ages researching this. When you try it out, as any sane person does, you don't get a nice "unsupported GPU" message from ROCm. The failure when you use ROCm is instability of your entire OS, not some clean crashing of the program you ran. This invites lots of messing around and desperately trying to get it to work and is a very frustrating experience and then all people do is say "look in this obscure GitHub issue they dropped support for you GPU, you're the one in the wrong".
- JonChesterfield 3y ago
  At least some of this is "support" meaning different things to different people. AMD means something like "your engineers can raise jira tickets related to the supercomputer you bought". The general idea outside of that context seems to be "support" means "can run stuff on the GPU". There's some analogy to games dev - if call of duty doesn't run on an amdgpu, you also don't get to complain to AMD about it, but the games studio certainly will do. The games studio is the entity with the contact details. ROCm is fairly likely to run on unsupported consumer cards if you build the libraries from source or use them from someone who did, which is just about viable from linux distributions now. You don't get to raise official jira tickets about it when it falls over because it's "unsupported". You'll have a moderately bad time if you ignore the docs recommendations about the kernel version for a given release, or if the difficult task of building the libraries from source goes awry. The established packaging idea of only shipping machine code libraries for a subset of the GPUs you sell, and giving very limited information on how to make those libraries exist for other GPUs, is obviously making essentially everyone very pissed off with AMD. Their engineers are aware of that and disagree about what to do about it. I personally think that'll resolve itself once Debian packaging is generally usable. People will install ROCm from their distro instead of from the official builds which don't have machine code for gaming GPUs anyway. That will give rise to new annoyances but at least the applications will try to run on whatever GPU you've got locally.
  - frognumber 3y ago
    "Support," to me, means not dropping support in newer versions of ROCm. See the current compatibility matrix: https://rocm.docs.amd.com/projects/radeon/en/latest/docs/compatibility.html https://rocm.docs.amd.com/projects/radeon/en/latest/docs/com... I bought my (in-production, advertised ROCm-capable) card during the great GPU shortage of 2020, and literally anything new stopped working a few months later when AMD decided to update ROCm to drop support for GFX8 GPUs: https://github.com/ROCm/ROCm/issues/1353 https://github.com/ROCm/ROCm/issues/1353 All the packages I was using (like Spacy) required later ROCm almost immediately. There was literally no way to make them work. Running older packages is a non-starter if you want things like, say, your machines to not be compromised. > if call of duty doesn't run on an amdgpu, you also don't get to complain to AMD about it, but the games studio certainly will do If "Call of Duty" is advertised on the box, and doesn't run, yes, this is false advertising, and "consumers" get to complain to the graphics card maker or AMD. In this case, small claims court would give me a refund if I sued AMD; an advertised ROCm-compatible GPU stopped being ROCm-compatible within the warranty period. It's just not worth complaining or suing over a few hundred bucks. I didn't need much performance, and this was purely for dev -- to make sure I bought this purely so I could have compatibility with AMD. AMD decided to not have compatibility with me, and I said f--k AMD. The result is that if you want to run my data pipeline, it won't work on AMD. A major industry will hopefully standardize on it, and guess who is locked out of selling any GPUs to that industry if that happens?
andrewstuart 3y ago
AMD need to fully support their cards as long as Nvidia does.

36 points by bx376 3y ago

I stopped supporting AMD when I bought an AMD card, and they discontinued support a few months later. If AMD sends me an MI300X for free, I'll make sure my software is compatible. Otherwise, if you want to run my stuff, you need Nvidia (or, hopefully soon, Intel). (Note: I know they never will. This story, multiplied by the number of people building stuff now, is why I'm bearish on AMD and bullish on competitors. This would be a very nice card -- 192GB is sexy -- but I just get headaches thinking about compatibility, software, and drivers issues. To quote Steve Balmer, what AMD is missing is: "Developers, developers, developers." That's not a small hole to dig out of.)

vlakreeh 3y ago
What card could you have bought that lost support months later? I can't think of any new card you could have bought that'd lose support that quickly as far back as 2016
- KeplerBoy 3y ago
  Not GP, but they dropped support for SKUs which were still sold at that point. The MI50 would be a fun accelerator to play with and easily worth 2000$ if it were made by Nvidia. Since they are made by AMD and not supported, you can grab them for 200$. https://github.com/ROCm/ROCm/issues/2308 https://github.com/ROCm/ROCm/issues/2308
  - simfree 3y ago
    Support still exists, but the card is in maintenance mode. This doesn't mean that your models that work on an MI50 today are suddenly going to stop working, or the models ran on my MI25 will cease suddenly.
    - frognumber 3y ago
      That's only true if you don't mind sharing your systems with strangers in Russia, Asia, and Africa. "Bug fixes / critical security patches will continue to be supported for the gfx906 GPUs till Q2 2024" Worse, a lot of tooling will break before then. If you need a security patch for anything built on top of ROCm or CUDA, you often can't have an ancient version. That's not to mention that I'd like to do new development and prototyping. I don't do business with AMD for the same reason I don't do business with Google. They're not a reliable partner.
      - nixgeek 3y ago
        That the only bad actors are in Russia, Asia and Africa is becoming quite the trope. Some very spectacular compromises by Lapsus$ aren’t originating from any of the above, but instead from the West, specifically the UK: https://www.bbc.com/news/technology-67663128 https://www.bbc.com/news/technology-67663128
        frognumber 3y ago
        I think you're missing the point. Bad actors are equally distributed around the world. The difference, however, is that, in this case, as the article points out, the bad actor is in custody. My local law enforcement, lawyer, and judicial system are able to help out with the bad actors here and keep the problem somewhat contained. For a bad actor in Russia or India, you quite literally have no recourse. There are literally Youtube videos (Mark Rober did a special recently) showing out-of-jurisdiction scammer and hacker shops occupying entire office buildings, and there is almost nothing we can do about it. I'm not even going to go on a diatribe about corruption and the quality of law enforcement, since one can have different opinions. What I will point out is that US+EU have extensive treaties which allow my local law enforcement agency to cooperate closely with law enforcement agencies between US, UK, and EU. For a bad actor in Iran or North Korea? You're not even getting diplomatic contact. Footnote: I mention India since they're a democracy and by all standards, try to act responsibly on the global stage. They just don't happen to be in-network for various US/EU-centric international conventions.
      - 3y ago
- frognumber 3y ago
  https://github.com/ROCm/ROCm/issues/1353 https://github.com/ROCm/ROCm/issues/1353 Bought in 2020. Stopped working in 2020. Not the latest, but in-production, advertised ROCm-capable, and what I could find during the Great GPU Shortage of 2020.
  - justinclift 3y ago
    Wow, not even an industry-standard-for-many-decades deprecation notice up front, to give people a heads up before hand. That's pretty fucked. :(
    - frognumber 3y ago
      It's even worse. Order of operations: 1. It just silently stopped working, started crashing, and I had no idea what was going on or why. Just sort of intermittent f-age where some older versions would work better, and newer ones would work worse. I never got it reliably working. In most cases, at some point, the system would become unresponsive, and then crash hard. 2. AMD removed it from the supported list (with no notice). 3. The github ticket above was filed, which explained what was going on. Lots of time wasted debugging. This interacted with a half-dozen other AMD bugs and issues (such as learning the GPU only worked for compute headless; I needed to drive my monitor with a different card). Human time is more expensive than equipment, so the total cost of this stuff was astronomical. Yes, notice is industry-standard, but at the very least, when support broke / was later removed, the driver could try warning me "We no longer support this card, do you REALLY want to proceed?" rather than letting me know by hard-crashing my system.
      - imtringued 3y ago
        People don't seem to understand how ROCm fails. Some inaccessible list somewhere buried deep down or a GitHub issues says your card is dropped. Apparently the average user is supposed to spend ages researching this. When you try it out, as any sane person does, you don't get a nice "unsupported GPU" message from ROCm. The failure when you use ROCm is instability of your entire OS, not some clean crashing of the program you ran. This invites lots of messing around and desperately trying to get it to work and is a very frustrating experience and then all people do is say "look in this obscure GitHub issue they dropped support for you GPU, you're the one in the wrong".
JonChesterfield 3y ago
At least some of this is "support" meaning different things to different people. AMD means something like "your engineers can raise jira tickets related to the supercomputer you bought". The general idea outside of that context seems to be "support" means "can run stuff on the GPU". There's some analogy to games dev - if call of duty doesn't run on an amdgpu, you also don't get to complain to AMD about it, but the games studio certainly will do. The games studio is the entity with the contact details. ROCm is fairly likely to run on unsupported consumer cards if you build the libraries from source or use them from someone who did, which is just about viable from linux distributions now. You don't get to raise official jira tickets about it when it falls over because it's "unsupported". You'll have a moderately bad time if you ignore the docs recommendations about the kernel version for a given release, or if the difficult task of building the libraries from source goes awry. The established packaging idea of only shipping machine code libraries for a subset of the GPUs you sell, and giving very limited information on how to make those libraries exist for other GPUs, is obviously making essentially everyone very pissed off with AMD. Their engineers are aware of that and disagree about what to do about it. I personally think that'll resolve itself once Debian packaging is generally usable. People will install ROCm from their distro instead of from the official builds which don't have machine code for gaming GPUs anyway. That will give rise to new annoyances but at least the applications will try to run on whatever GPU you've got locally.
- frognumber 3y ago
  "Support," to me, means not dropping support in newer versions of ROCm. See the current compatibility matrix: https://rocm.docs.amd.com/projects/radeon/en/latest/docs/compatibility.html https://rocm.docs.amd.com/projects/radeon/en/latest/docs/com... I bought my (in-production, advertised ROCm-capable) card during the great GPU shortage of 2020, and literally anything new stopped working a few months later when AMD decided to update ROCm to drop support for GFX8 GPUs: https://github.com/ROCm/ROCm/issues/1353 https://github.com/ROCm/ROCm/issues/1353 All the packages I was using (like Spacy) required later ROCm almost immediately. There was literally no way to make them work. Running older packages is a non-starter if you want things like, say, your machines to not be compromised. > if call of duty doesn't run on an amdgpu, you also don't get to complain to AMD about it, but the games studio certainly will do If "Call of Duty" is advertised on the box, and doesn't run, yes, this is false advertising, and "consumers" get to complain to the graphics card maker or AMD. In this case, small claims court would give me a refund if I sued AMD; an advertised ROCm-compatible GPU stopped being ROCm-compatible within the warranty period. It's just not worth complaining or suing over a few hundred bucks. I didn't need much performance, and this was purely for dev -- to make sure I bought this purely so I could have compatibility with AMD. AMD decided to not have compatibility with me, and I said f--k AMD. The result is that if you want to run my data pipeline, it won't work on AMD. A major industry will hopefully standardize on it, and guess who is locked out of selling any GPUs to that industry if that happens?

andrewstuart 3y ago

AMD need to fully support their cards as long as Nvidia does.