7 ms·

I was able to use ROCm recently with Pytorch and after pulling some hair it worked quite well. The Radeon GPU I had on hand was a bit old and underpowered (RDNA

I was able to use ROCm recently with Pytorch and after pulling some hair it worked quite well. The Radeon GPU I had on hand was a bit old and underpowered (RDNA2) and it only supported matmul on fp64, but for the job I needed done I saw a 200x increase in it/s over CPU despite the need to cast everywhere, and that made me super happy. Best of all is that I simply set the device to `torch.device('cuda')` rather than openCL, which does wonders for compatibility and to keep code simple. Protip: Use the official ROCM Pytorch base docker image [0]. The AMD setup is so finicky and dependent on specific versions of sdk/drivers/libraries and it will be much harder to make work if you try to install them separately. [0]: https://rocm.docs.amd.com/en/latest/how_to/pytorch_install/pytorch_install.html https://rocm.docs.amd.com/en/latest/how_to/pytorch_install/p...

mikepurvis 3y ago
Sigh. It's great that these container images exist to give people an easy on-ramp, but they definitely don't work for every use case (especially once you're in embedded where space matters and you might not be online to pull multi-gb updates from some registry). So it's important that vendors don't feel let off the hook to provide sane packaging just because there's an option to use a kitchen-sink container image they rebuild every day from source.
- fwsgonzo 3y ago
  I feel the same way, especially about build systems. OpenSSL and v8 are among a large list of things that have horrid build systems. Only way to build them sanely is to use some randos CMake fork, then it Just Works. Literally a two-liner in your build system to add them to your project with a sane CMake script.
  - mikepurvis 3y ago
    I was part of a Nix migration over the past two years, and literally one of the first things we checked is that there was already a community-maintained tensorflow+gpu package in nixpkgs because without that the whole thing would have been a complete non-starter, and we sure as heck didn't have the resources or know-how to figure it out for ourselves as a small DevOps team just trying to do basic packaging.
- mathisfun123 3y ago
  > especially once you're in embedded is this a real problem? exactly which embedded platform has a device that ROCm supports?
  - mikepurvis 3y ago
    Robotic perception is the one relevant to me. You want to do object recognition on an industrial x86 or Jetson-type machine, without having to use Ubuntu or whatever the one "blessed" underlay system is (either natively or implicitly because you pulled a container based on it).
    - mathisfun123 3y ago
      >industrial x86 or Jetson-type machine that's not embedded dev. if you 1. use underpowered devices to perform sophisticated tasks 2. using code/tools that operate at extremely high levels of "abstraction" don't be surprised when all the inherent complexity is tamed using just more layers of "abstraction". if that becomes a problem for your cost/power/space budget then reconsider choice 1 or choice 2.
      - mikepurvis 3y ago
        Not sure this is worth an argument over semantics, but modern "embedded" development is a lot bigger than just microcontrollers and wearables. IMO as soon as you're deploying a computer into any kind of "appliance", or you're offline for periods of time, or you're running on batteries or your primary network connection is wireless... then yeah, you're starting to hit the requirements associated with embedded and need to seek established solutions for them, including using distros which account for those requirements.
        mathisfun123 3y ago
        > IMO as soon as you're deploying a computer into any kind of "appliance", or you're offline for periods of time, or you're running on batteries or your primary network connection is wireless yes and in those instances you do not reach for pytorch/tensorflow on top of ubuntu on top of x86 with a discrete gpu and 32gb of ram. instead you reach for C and micro or some arm soc that supports baremetal or at most rtos. that's embedded dev. so i'll repeat myself: if you want to run extremely high-level code then don't be "surprised pikachu" when your underpowered platform, that you chose due to concrete, tight budgets doesn't work out.
- xahrepap 3y ago
  I know it's still different than what you're looking for, so you probably already know this, but many projects like this have the Dockerfile on github which shows exactly how they set up the image. For example: https://github.com/RadeonOpenCompute/ROCm-docker/blob/master/dev/Dockerfile-ubuntu-22.04 https://github.com/RadeonOpenCompute/ROCm-docker/blob/master... They also have some for Fedora. Looks like for this you need to install their repo: curl -sL https://repo.radeon.com/rocm/rocm.gpg.key | apt-key add - \ && printf "deb [arch=amd64] https://repo.radeon.com/rocm/apt/$ROCM_VERSION/ jammy main" | tee /etc/apt/sources.list.d/rocm.list \ && printf "deb [arch=amd64] https://repo.radeon.com/amdgpu/$AMDGPU_VERSION/ubuntu jammy main" | tee /etc/apt/sources.list.d/amdgpu.list \ then install Python, a couple other dependencies (build-essential, etc) and then the package in question: rocm-dev So they are doing the packaging. There might even be documentation elsewhere for that type of setup.
  - mikepurvis 3y ago
    Oh yeah, I mean... having the source for the container build is kind of table stakes at this point. No one would accept a 10gb mystery meat blob as the basis of their production system. It's bad enough that we still accept binary-only drivers and proprietary libraries like TensorRT. I think my issue is more just with the mindset that it's okay to have one narrow slice of supported versions of everything that are "known to work together" and those are what's in the container and anything outside of those and you're immediately pooched. This is not hypothetical btw, I've run into real problems around it with libraries like gproto, where tensorflow's bazel build pulls in an exact version that's different from the default one in nixpkgs, and now you get symbol conflicts when something tries to link to the tensorflow c++ API while linking to another component already using the default gproto. I know these problems are solveable with symbol visibility control and whatever, but that stuff is far from universal and hard to get right, especially if the person setting up the build rules for the library doesn't themselves use it in that type of heterogeneous environment (like, everyone at Google just links the same global proto version from the monorepo so it doesn't matter).
    - hotstickyballs 3y ago
      If anything, the situation with tensor rt shows that companies are absolutely willing to accept a multi gig meat blob
- amelius 3y ago
  > So it's important that vendors don't feel let off the hook to provide sane packaging just because there's an option to use a kitchen-sink container image they rebuild every day. Sadly if e.g. 95% of their users can use the container, then it could make economical sense to do it that way.
- ngcc_hk 3y ago
  Better to come if the tide shift so we can have compatible layer. The key is the tide. Obviously would n try to sue … it would be a sign that finally we have real competition. Gar is where innovation do. X86 cannot do 64 bit let us do this and that so the market can use only our cpu. Repeat with me x86-64 is impossible. Not sure Apple is in this otherwise the real great competition come.
wyldfire 3y ago
> Best of all is that I simply set the device to `torch.device('cuda')` rather than openCL, which does wonders for compatibility Man oh man where did we go wrong that cuda is the more compatible option over OpenCL?
- NavinF 3y ago
  This has always been the case. OpenCL is a shit show
- KeplerBoy 3y ago
  It must be a misnomer on PyTorch's side. Clearly it's neither CUDA nor OpenCL. AMD should just get it's shit together. This is ridiculous. Not the name, but the fact that you can only do FP64 on a GPU. Everybody is moving to FP16 and AMD is stuck on doubles?
  - omneity 3y ago
    I believe the fp64 limitation came from the laptop-grade GPU I had rather than inherent to AMD or ROCm. The API level I could target was at least two or three versions behind the latest they have to offer.
    - KeplerBoy 3y ago
      Might very well be true. I don't blame anyone for not diving deeper into figuring out why this stuff doesn't work. But this is one of the great strengths of CUDA: I can develop a kernel on my workstation, my boss can demo it on his laptop and we can deploy it on Jetsons or the multi-gpu cluster with minimal changes and i can be sure that everything runs everywhere.
      - brutus1213 3y ago
        There is indeed something excellent about CUDA from a user perspective that is hard to beat. I do high-level DNN and it is not clear to me what it is or why that is. Anytime I have worked on optimizing to mobile hardware (not Jetson, but actual phones or accelerators), it is just a world of hurt and incompatibilities. This notion that operators or subgraphs can be accelerated by lower level closed blobs .. I wonder if that is part of the issue. But then why doesn't OpenCL not just work? I thought it gave a CUDA kernel like general purpose abstraction. I just don't understand the details enough to understand why things are problematic without CUDA :(
RockRobotRock 3y ago
Have you gotten it to work with Whisper by any chance?
- kkielhofner 3y ago
  Whisper is actually a great example of why Nvidia has such a stronghold on ML/AI and why it’s so difficult to compete. There’s getting something to “work”, which is often enough of a challenge with ROCm. Then there’s getting it to work well (next challenge). Then there’s getting it to work as well as Nvidia/CUDA. With Whisper, as one example, you should be running it with ctranslate2[0]. Of all the platforms on their supported list you won’t find ROCm. When you really start to look around you’ll find that ROCm is (at best) still very much in the “get it to work (sometimes)” stage. In most cases it’s still a long way away from getting it to work well, and even further away from making it actually competitive with Nvidia for serious use cases and applications. People get excited about the progress ROCm has made getting basic things to work with PyTorch and this is good - progress is progress. But saving 20% on the hardware when the equivalent Nvidia product is often somewhere between 5-10x as performant (at a fraction of the development time) because of vastly superior software support you realize pretty quickly Nvidia is actually a bargain compared to AMD. I’m desperately rooting for Nvidia to have some actual competition but after six years of ROCm and my own repeated failed attempts to have it make any sense overall I’m only more and more skeptical that real competition in the space will come from AMD. [0] - https://github.com/OpenNMT/CTranslate2 https://github.com/OpenNMT/CTranslate2
  - errnoh 3y ago
    While I agree that it's much more effort to get things working on AMD cards than it is with Nvidia, I was a bit surprised to see this comment mention Whisper being an example of "5-10x as performant". https://www.tomshardware.com/news/whisper-audio-transcription-gpus-benchmarked https://www.tomshardware.com/news/whisper-audio-transcriptio... is a good example of Nvidia having no excuses being double the price when it comes to Whisper inference, with 7900XTX being directly comparable with 4080, albeit with higher power draw. To be fair it's not using ROCm but Direct3D 11, but for performance/price arguments sake that detail is not relevant. EDIT: Also using CTranslate2 as an example is not great as it's actually a good showcase why ROCm is so far behind CUDA: It's all about adapting the tech and getting the popular libraries to support it. Things usually get implemented in CUDA first and then would need additional effort to add ROCm support that projects with low amount of (possibly hobbyist) maintainers might not have available. There's even an issue in CTranslate2 where they clearly state no-one is working to get ROCm supported in the library. ( https://github.com/OpenNMT/CTranslate2/issues/1072#issuecomment-1696983435 https://github.com/OpenNMT/CTranslate2/issues/1072#issuecomm... )
incognition 3y ago
Fp64??
- latchkey 3y ago
  https://en.wikipedia.org/wiki/Double-precision_floating-point_format https://en.wikipedia.org/wiki/Double-precision_floating-poin... NVIDIA fp32 (H100) has 2x more TFLOPS than AMD's fp32 (MI250) and AI doesn't need fp64 precision.
  - incognition 3y ago
    Lol it was meant as I wouldn't be caught dead using fp64
- fransje26 3y ago
  Hardware limitation.

mikepurvis 3y ago

Sigh. It's great that these container images exist to give people an easy on-ramp, but they definitely don't work for every use case (especially once you're in embedded where space matters and you might not be online to pull multi-gb updates from some registry). So it's important that vendors don't feel let off the hook to provide sane packaging just because there's an option to use a kitchen-sink container image they rebuild every day from source.

fwsgonzo 3y ago
I feel the same way, especially about build systems. OpenSSL and v8 are among a large list of things that have horrid build systems. Only way to build them sanely is to use some randos CMake fork, then it Just Works. Literally a two-liner in your build system to add them to your project with a sane CMake script.
- mikepurvis 3y ago
  I was part of a Nix migration over the past two years, and literally one of the first things we checked is that there was already a community-maintained tensorflow+gpu package in nixpkgs because without that the whole thing would have been a complete non-starter, and we sure as heck didn't have the resources or know-how to figure it out for ourselves as a small DevOps team just trying to do basic packaging.
mathisfun123 3y ago
> especially once you're in embedded is this a real problem? exactly which embedded platform has a device that ROCm supports?
- mikepurvis 3y ago
  Robotic perception is the one relevant to me. You want to do object recognition on an industrial x86 or Jetson-type machine, without having to use Ubuntu or whatever the one "blessed" underlay system is (either natively or implicitly because you pulled a container based on it).
  - mathisfun123 3y ago
    >industrial x86 or Jetson-type machine that's not embedded dev. if you 1. use underpowered devices to perform sophisticated tasks 2. using code/tools that operate at extremely high levels of "abstraction" don't be surprised when all the inherent complexity is tamed using just more layers of "abstraction". if that becomes a problem for your cost/power/space budget then reconsider choice 1 or choice 2.
    - mikepurvis 3y ago
      Not sure this is worth an argument over semantics, but modern "embedded" development is a lot bigger than just microcontrollers and wearables. IMO as soon as you're deploying a computer into any kind of "appliance", or you're offline for periods of time, or you're running on batteries or your primary network connection is wireless... then yeah, you're starting to hit the requirements associated with embedded and need to seek established solutions for them, including using distros which account for those requirements.
      - mathisfun123 3y ago
        > IMO as soon as you're deploying a computer into any kind of "appliance", or you're offline for periods of time, or you're running on batteries or your primary network connection is wireless yes and in those instances you do not reach for pytorch/tensorflow on top of ubuntu on top of x86 with a discrete gpu and 32gb of ram. instead you reach for C and micro or some arm soc that supports baremetal or at most rtos. that's embedded dev. so i'll repeat myself: if you want to run extremely high-level code then don't be "surprised pikachu" when your underpowered platform, that you chose due to concrete, tight budgets doesn't work out.
xahrepap 3y ago
I know it's still different than what you're looking for, so you probably already know this, but many projects like this have the Dockerfile on github which shows exactly how they set up the image. For example: https://github.com/RadeonOpenCompute/ROCm-docker/blob/master/dev/Dockerfile-ubuntu-22.04 https://github.com/RadeonOpenCompute/ROCm-docker/blob/master... They also have some for Fedora. Looks like for this you need to install their repo: curl -sL https://repo.radeon.com/rocm/rocm.gpg.key | apt-key add - \ && printf "deb [arch=amd64] https://repo.radeon.com/rocm/apt/$ROCM_VERSION/ jammy main" | tee /etc/apt/sources.list.d/rocm.list \ && printf "deb [arch=amd64] https://repo.radeon.com/amdgpu/$AMDGPU_VERSION/ubuntu jammy main" | tee /etc/apt/sources.list.d/amdgpu.list \ then install Python, a couple other dependencies (build-essential, etc) and then the package in question: rocm-dev So they are doing the packaging. There might even be documentation elsewhere for that type of setup.
- mikepurvis 3y ago
  Oh yeah, I mean... having the source for the container build is kind of table stakes at this point. No one would accept a 10gb mystery meat blob as the basis of their production system. It's bad enough that we still accept binary-only drivers and proprietary libraries like TensorRT. I think my issue is more just with the mindset that it's okay to have one narrow slice of supported versions of everything that are "known to work together" and those are what's in the container and anything outside of those and you're immediately pooched. This is not hypothetical btw, I've run into real problems around it with libraries like gproto, where tensorflow's bazel build pulls in an exact version that's different from the default one in nixpkgs, and now you get symbol conflicts when something tries to link to the tensorflow c++ API while linking to another component already using the default gproto. I know these problems are solveable with symbol visibility control and whatever, but that stuff is far from universal and hard to get right, especially if the person setting up the build rules for the library doesn't themselves use it in that type of heterogeneous environment (like, everyone at Google just links the same global proto version from the monorepo so it doesn't matter).
  - hotstickyballs 3y ago
    If anything, the situation with tensor rt shows that companies are absolutely willing to accept a multi gig meat blob
amelius 3y ago
> So it's important that vendors don't feel let off the hook to provide sane packaging just because there's an option to use a kitchen-sink container image they rebuild every day. Sadly if e.g. 95% of their users can use the container, then it could make economical sense to do it that way.
ngcc_hk 3y ago
Better to come if the tide shift so we can have compatible layer. The key is the tide. Obviously would n try to sue … it would be a sign that finally we have real competition. Gar is where innovation do. X86 cannot do 64 bit let us do this and that so the market can use only our cpu. Repeat with me x86-64 is impossible. Not sure Apple is in this otherwise the real great competition come.

wyldfire 3y ago

> Best of all is that I simply set the device to `torch.device('cuda')` rather than openCL, which does wonders for compatibility Man oh man where did we go wrong that cuda is the more compatible option over OpenCL?

NavinF 3y ago
This has always been the case. OpenCL is a shit show
KeplerBoy 3y ago
It must be a misnomer on PyTorch's side. Clearly it's neither CUDA nor OpenCL. AMD should just get it's shit together. This is ridiculous. Not the name, but the fact that you can only do FP64 on a GPU. Everybody is moving to FP16 and AMD is stuck on doubles?
- omneity 3y ago
  I believe the fp64 limitation came from the laptop-grade GPU I had rather than inherent to AMD or ROCm. The API level I could target was at least two or three versions behind the latest they have to offer.
  - KeplerBoy 3y ago
    Might very well be true. I don't blame anyone for not diving deeper into figuring out why this stuff doesn't work. But this is one of the great strengths of CUDA: I can develop a kernel on my workstation, my boss can demo it on his laptop and we can deploy it on Jetsons or the multi-gpu cluster with minimal changes and i can be sure that everything runs everywhere.
    - brutus1213 3y ago
      There is indeed something excellent about CUDA from a user perspective that is hard to beat. I do high-level DNN and it is not clear to me what it is or why that is. Anytime I have worked on optimizing to mobile hardware (not Jetson, but actual phones or accelerators), it is just a world of hurt and incompatibilities. This notion that operators or subgraphs can be accelerated by lower level closed blobs .. I wonder if that is part of the issue. But then why doesn't OpenCL not just work? I thought it gave a CUDA kernel like general purpose abstraction. I just don't understand the details enough to understand why things are problematic without CUDA :(

RockRobotRock 3y ago

Have you gotten it to work with Whisper by any chance?

kkielhofner 3y ago
Whisper is actually a great example of why Nvidia has such a stronghold on ML/AI and why it’s so difficult to compete. There’s getting something to “work”, which is often enough of a challenge with ROCm. Then there’s getting it to work well (next challenge). Then there’s getting it to work as well as Nvidia/CUDA. With Whisper, as one example, you should be running it with ctranslate2[0]. Of all the platforms on their supported list you won’t find ROCm. When you really start to look around you’ll find that ROCm is (at best) still very much in the “get it to work (sometimes)” stage. In most cases it’s still a long way away from getting it to work well, and even further away from making it actually competitive with Nvidia for serious use cases and applications. People get excited about the progress ROCm has made getting basic things to work with PyTorch and this is good - progress is progress. But saving 20% on the hardware when the equivalent Nvidia product is often somewhere between 5-10x as performant (at a fraction of the development time) because of vastly superior software support you realize pretty quickly Nvidia is actually a bargain compared to AMD. I’m desperately rooting for Nvidia to have some actual competition but after six years of ROCm and my own repeated failed attempts to have it make any sense overall I’m only more and more skeptical that real competition in the space will come from AMD. [0] - https://github.com/OpenNMT/CTranslate2 https://github.com/OpenNMT/CTranslate2
- errnoh 3y ago
  While I agree that it's much more effort to get things working on AMD cards than it is with Nvidia, I was a bit surprised to see this comment mention Whisper being an example of "5-10x as performant". https://www.tomshardware.com/news/whisper-audio-transcription-gpus-benchmarked https://www.tomshardware.com/news/whisper-audio-transcriptio... is a good example of Nvidia having no excuses being double the price when it comes to Whisper inference, with 7900XTX being directly comparable with 4080, albeit with higher power draw. To be fair it's not using ROCm but Direct3D 11, but for performance/price arguments sake that detail is not relevant. EDIT: Also using CTranslate2 as an example is not great as it's actually a good showcase why ROCm is so far behind CUDA: It's all about adapting the tech and getting the popular libraries to support it. Things usually get implemented in CUDA first and then would need additional effort to add ROCm support that projects with low amount of (possibly hobbyist) maintainers might not have available. There's even an issue in CTranslate2 where they clearly state no-one is working to get ROCm supported in the library. ( https://github.com/OpenNMT/CTranslate2/issues/1072#issuecomment-1696983435 https://github.com/OpenNMT/CTranslate2/issues/1072#issuecomm... )

incognition 3y ago

Fp64??

latchkey 3y ago
https://en.wikipedia.org/wiki/Double-precision_floating-point_format https://en.wikipedia.org/wiki/Double-precision_floating-poin... NVIDIA fp32 (H100) has 2x more TFLOPS than AMD's fp32 (MI250) and AI doesn't need fp64 precision.
- incognition 3y ago
  Lol it was meant as I wouldn't be caught dead using fp64
fransje26 3y ago
Hardware limitation.