14 ms·

ROCm is not the only option, compute shaders are very reliable on all GPUs. And thanks to the Valve’s work on DXVK 2.0, modern Linux runs Windows D3D11 software

ROCm is not the only option, compute shaders are very reliable on all GPUs. And thanks to the Valve’s work on DXVK 2.0, modern Linux runs Windows D3D11 software just fine. Here’s an example https://github.com/Const-me/Whisper/issues/42 https://github.com/Const-me/Whisper/issues/42 BTW, a lot of BLAS in the compute shaders of that software.

roenxi 3y ago
I dunno, are they? AMD should to pay someone to put up some "how to multiply a 2x2 matrix on our GPU for the average programmer!" tutorials somewhere obvious. I saw a lot of GPU lockups before I gave up on trying and decided that it wasn't worth it. Maybe compute shaders were a thing I should have tried. To be honest, I don't know much about them because my attempts in the space were shut down pretty hard by driver bugs linked to OpenCL and ROCm. I thought it was just me for a while, but after watching George Hotz's famous meltdown trying to program on an AMD GPU I do wonder if they're underestimating the power of a few good public "how to use the damn thing" sessions. They've been pushing ROCm which would probably be great if it worked reliably.
- Const-me 3y ago
  > driver bugs linked to OpenCL and ROCm CUDA is the default tech for GPGPU in HPC or AI applications, for more than a decade now. By now, people have found most of these driver bugs, and nVidia has fixed them. Similarly, compute shaders is the only tech for GPGPU used in videogames. Modern videogames are using compute shaders for a decade now, in increasing volumes. For example, UE5 even renders triangle meshes with them [1]. However, OpenCL and ROCm are niche technologies. I’ve been hearing complaints about driver quality for some time now. For obvious reason, AMD and Intel prioritize driver bugs which affect modern videogames sold in many million copies, compared to the bugs which only affect a few people working on HPC, AI or other niche GPGPU applications. > they're underestimating the power of a few good public "how to use the damn thing" sessions I agree the learning curve is steep, with the lack of good materials. For an introduction article, see [2]. Ignore the parts about D3D10 hardware, the article is old and D3D10 hardware is no longer relevant. Another one, with slightly more depth, is [3]. For an example how to multiply large dense matrices with a compute shader see [4], but that example is rather advanced because optimizations, and because weird memory layout conventions inherited from the upstream project. [1] https://www.youtube.com/watch?v=TMorJX3Nj6U https://www.youtube.com/watch?v=TMorJX3Nj6U [2] https://developer.download.nvidia.com/compute/DevZone/docs/html/DirectCompute/doc/DirectCompute_Programming_Guide.pdf https://developer.download.nvidia.com/compute/DevZone/docs/h... [3] https://github.com/jstoecker/dxcompute-docs/tree/main https://github.com/jstoecker/dxcompute-docs/tree/main [4] https://github.com/Const-me/Whisper/blob/master/ComputeShaders/mulMatTiled.hlsl https://github.com/Const-me/Whisper/blob/master/ComputeShade...
  - mschuster91 3y ago
    > For obvious reason, AMD and Intel prioritize driver bugs which affect modern videogames sold in many million copies, compared to the bugs which only affect a few people working on HPC, AI or other niche GPGPU applications. If people could develop AI stuff on consumer cards, they'd buy a ton of server-grade cards, or rent them via the usual hyperscalers or dedicated platforms for the actual work. This entire, multi-million dollar (per training session) market is all firmly in the hands of nVIDIA at the moment, and unless AMD seriously improves their offering that won't change. nVIDIA got to the point they are because they focused on getting developers up to speed very fast and cheap, and so the developers asked their employers to get them the stuff they were already used to.
    - Const-me 3y ago
      > If people could develop AI stuff on consumer cards Technically people can already do that, by leveraging compute shaders in D3D, Vulkan, or WebGPU. I’m certain it’s possible to implement D3D or Vulkan backend for PyTorch, TensorFlow, and similar ML libraries. When I experimented with AI, I did similar stuff myself with C++, C#, and HLSL, and I found it wasn’t terribly hard to do. However, PyTorch made by Facebook, TensorFlow by Google. It seems most companies who maintain such libraries are only interested in cloud computing. Some of them, like Qualcomm, only care about their own proprietary hardware. None of them seems to care about desktops.
    - mrguyorama 3y ago
      My theory is that AMD initially wanted to compete in compute with nVidia and was going to improve ROCm but then they saw how much trouble nVidia had with artificially keeping enterprise users from just buying consumer GPUs instead of the much more profitable enterprise GPUs so they trashed that idea to keep consumer GPUs from interfering with their very profitable enterprise GPU business for compute applications.
      - JonChesterfield 3y ago
        Rocm was designed and implemented for HPC. There's no cunning scheme to stop it working on gaming cards, there just isn't (wasn't?) much investment in making it work either.
singhrac 3y ago
Sure, compute shaders might work, but don’t you need rocBLAS, rocSPARSE, MIOpen, etc? Are people reinventing those in compute shaders in another package?
- Const-me 3y ago
  These things are nice to have, but you don’t actually need them. It only takes 1-2 pages of HLSL to implement efficient matrix multiplication. It’s not rocket science, the area is well researched and there’re many good articles on how to implement these BLAS routines efficiently. Moreover, manually written compute shaders enable stuff missing from these off-the-shelf higher level libraries. It’s easy to merge multiple compute operations into a single shader. When possible, this sometimes saves gigabytes of memory bandwidth (and therefore time) these high-level libraries spend writing/reading temporary tensors. It’s possible to re-shape immutable or rarely changing tensors into better memory layouts. Example for CPU compute https://stackoverflow.com/a/75567894/126995 https://stackoverflow.com/a/75567894/126995 the idea is equally good on GPUs. It’s possible to use custom data formats, and they don’t require any hardware support. Upcasting BF16 to FP32 is 1 shader instruction (left shift), downcasting FP32 to BF16 only takes a few of them (for proper rounding), no hardware support necessary. Can pack quantized or sparse tensors into a single ByteAddressBuffer, again nothing special is required from hardware. Can implement custom compression formats for these tensors.
  - empyrrhicist 3y ago
    So everyone that wants to use GPUs to accelerate their compute needs to write a BLAS implementation in shaders... FFS that's not reasonable at all! Sure, it's entirely possible, but there's a reason that high level libraries are the go-to for scientific compute.
  - kiratp 3y ago
    The bar for the AI ecosystem to get models running is git clone + <20 lines of Python. If AMD can’t make that work on consumer GPUs, then the only option they have is to undercut Nvidia so deeply that it makes sense for the big players to hire a team to write AMD’s software for them.
dragontamer 3y ago
It disappoints me that DirectX remains one of the best GPU-compute solutions in practice right now. And Vulkan too I guess. But it really is. That's the state of the market. The video game artists are GPU-programmers, they've hit DirectX11, DX12, and Vulkan with a wide variety of video games and have turned that ecosystem very stable. ------------- DX11 is 32-bit only atomics, I don't think its a very serious solution in practice. 64-bit atomics (especially 64-bit CAS) is already very limiting compared to CPU-world where 128-bit CAS is needed to fix the obscure ABA-problem. DX and Vulkan just have... so much API-crap you need to even get Hello World / SAXY up. C++Amp was wonderful back in 2014, but it too is stuck in DirectX11 and therefore 32-bit atomic world. And it hasn't had an update since then. Microsoft really should have kept investing in C++Amp IMO. ------------- ROCm is fine if you get the hardware and if it remains supported. But I think in practice, people expect support longer than what AMD is willing to give.
- Const-me 3y ago
  > 32-bit only atomics, I don't think its a very serious solution in practice Yeah, I think I encountered that while porting a hash map from CUDA to HLSL. However, I'm not sure that's necessarily a huge deal. Probably not an issue for machine learning or BLAS stuff, these use cases don't need fine-grained thread synchronization. For applications which would benefit from such synchronization, traditional lock-free techniques ported from CPU (i.e. compare and swap atomics on global memory) can be slow due to huge count of active threads on GPUs. I mean it's obviously better than locks, but sometimes it's possible to do something better instead of CAS. > so much API-crap you need to even get Hello World / SAXY up. I agree for D3D12 and especially Vulkan, but IMO D3D11 is not terribly bad. It has a steep learning curve, but the amount of boilerplate for simple apps is IMO reasonable. Especially for ML or similar GPGPU stuff which only needs a small subset of the API: compute shaders only, no textures, no render targets, no depth-stencil views, no input layouts, etc. However, unlike simple apps, real-life ones often need profiler and queue depth limiter, relatively hard to implement on top of these queries. I think Microsoft should ship them both in Windows SDK.
  - dragontamer 3y ago

roenxi 3y ago

I dunno, are they? AMD should to pay someone to put up some "how to multiply a 2x2 matrix on our GPU for the average programmer!" tutorials somewhere obvious. I saw a lot of GPU lockups before I gave up on trying and decided that it wasn't worth it. Maybe compute shaders were a thing I should have tried. To be honest, I don't know much about them because my attempts in the space were shut down pretty hard by driver bugs linked to OpenCL and ROCm. I thought it was just me for a while, but after watching George Hotz's famous meltdown trying to program on an AMD GPU I do wonder if they're underestimating the power of a few good public "how to use the damn thing" sessions. They've been pushing ROCm which would probably be great if it worked reliably.

Const-me 3y ago
> driver bugs linked to OpenCL and ROCm CUDA is the default tech for GPGPU in HPC or AI applications, for more than a decade now. By now, people have found most of these driver bugs, and nVidia has fixed them. Similarly, compute shaders is the only tech for GPGPU used in videogames. Modern videogames are using compute shaders for a decade now, in increasing volumes. For example, UE5 even renders triangle meshes with them [1]. However, OpenCL and ROCm are niche technologies. I’ve been hearing complaints about driver quality for some time now. For obvious reason, AMD and Intel prioritize driver bugs which affect modern videogames sold in many million copies, compared to the bugs which only affect a few people working on HPC, AI or other niche GPGPU applications. > they're underestimating the power of a few good public "how to use the damn thing" sessions I agree the learning curve is steep, with the lack of good materials. For an introduction article, see [2]. Ignore the parts about D3D10 hardware, the article is old and D3D10 hardware is no longer relevant. Another one, with slightly more depth, is [3]. For an example how to multiply large dense matrices with a compute shader see [4], but that example is rather advanced because optimizations, and because weird memory layout conventions inherited from the upstream project. [1] https://www.youtube.com/watch?v=TMorJX3Nj6U https://www.youtube.com/watch?v=TMorJX3Nj6U [2] https://developer.download.nvidia.com/compute/DevZone/docs/html/DirectCompute/doc/DirectCompute_Programming_Guide.pdf https://developer.download.nvidia.com/compute/DevZone/docs/h... [3] https://github.com/jstoecker/dxcompute-docs/tree/main https://github.com/jstoecker/dxcompute-docs/tree/main [4] https://github.com/Const-me/Whisper/blob/master/ComputeShaders/mulMatTiled.hlsl https://github.com/Const-me/Whisper/blob/master/ComputeShade...
- mschuster91 3y ago
  > For obvious reason, AMD and Intel prioritize driver bugs which affect modern videogames sold in many million copies, compared to the bugs which only affect a few people working on HPC, AI or other niche GPGPU applications. If people could develop AI stuff on consumer cards, they'd buy a ton of server-grade cards, or rent them via the usual hyperscalers or dedicated platforms for the actual work. This entire, multi-million dollar (per training session) market is all firmly in the hands of nVIDIA at the moment, and unless AMD seriously improves their offering that won't change. nVIDIA got to the point they are because they focused on getting developers up to speed very fast and cheap, and so the developers asked their employers to get them the stuff they were already used to.
  - Const-me 3y ago
    > If people could develop AI stuff on consumer cards Technically people can already do that, by leveraging compute shaders in D3D, Vulkan, or WebGPU. I’m certain it’s possible to implement D3D or Vulkan backend for PyTorch, TensorFlow, and similar ML libraries. When I experimented with AI, I did similar stuff myself with C++, C#, and HLSL, and I found it wasn’t terribly hard to do. However, PyTorch made by Facebook, TensorFlow by Google. It seems most companies who maintain such libraries are only interested in cloud computing. Some of them, like Qualcomm, only care about their own proprietary hardware. None of them seems to care about desktops.
  - mrguyorama 3y ago
    My theory is that AMD initially wanted to compete in compute with nVidia and was going to improve ROCm but then they saw how much trouble nVidia had with artificially keeping enterprise users from just buying consumer GPUs instead of the much more profitable enterprise GPUs so they trashed that idea to keep consumer GPUs from interfering with their very profitable enterprise GPU business for compute applications.
    - JonChesterfield 3y ago
      Rocm was designed and implemented for HPC. There's no cunning scheme to stop it working on gaming cards, there just isn't (wasn't?) much investment in making it work either.

singhrac 3y ago

Sure, compute shaders might work, but don’t you need rocBLAS, rocSPARSE, MIOpen, etc? Are people reinventing those in compute shaders in another package?

Const-me 3y ago
These things are nice to have, but you don’t actually need them. It only takes 1-2 pages of HLSL to implement efficient matrix multiplication. It’s not rocket science, the area is well researched and there’re many good articles on how to implement these BLAS routines efficiently. Moreover, manually written compute shaders enable stuff missing from these off-the-shelf higher level libraries. It’s easy to merge multiple compute operations into a single shader. When possible, this sometimes saves gigabytes of memory bandwidth (and therefore time) these high-level libraries spend writing/reading temporary tensors. It’s possible to re-shape immutable or rarely changing tensors into better memory layouts. Example for CPU compute https://stackoverflow.com/a/75567894/126995 https://stackoverflow.com/a/75567894/126995 the idea is equally good on GPUs. It’s possible to use custom data formats, and they don’t require any hardware support. Upcasting BF16 to FP32 is 1 shader instruction (left shift), downcasting FP32 to BF16 only takes a few of them (for proper rounding), no hardware support necessary. Can pack quantized or sparse tensors into a single ByteAddressBuffer, again nothing special is required from hardware. Can implement custom compression formats for these tensors.
- empyrrhicist 3y ago
  So everyone that wants to use GPUs to accelerate their compute needs to write a BLAS implementation in shaders... FFS that's not reasonable at all! Sure, it's entirely possible, but there's a reason that high level libraries are the go-to for scientific compute.
- kiratp 3y ago
  The bar for the AI ecosystem to get models running is git clone + <20 lines of Python. If AMD can’t make that work on consumer GPUs, then the only option they have is to undercut Nvidia so deeply that it makes sense for the big players to hire a team to write AMD’s software for them.

dragontamer 3y ago

It disappoints me that DirectX remains one of the best GPU-compute solutions in practice right now. And Vulkan too I guess. But it really is. That's the state of the market. The video game artists are GPU-programmers, they've hit DirectX11, DX12, and Vulkan with a wide variety of video games and have turned that ecosystem very stable. ------------- DX11 is 32-bit only atomics, I don't think its a very serious solution in practice. 64-bit atomics (especially 64-bit CAS) is already very limiting compared to CPU-world where 128-bit CAS is needed to fix the obscure ABA-problem. DX and Vulkan just have... so much API-crap you need to even get Hello World / SAXY up. C++Amp was wonderful back in 2014, but it too is stuck in DirectX11 and therefore 32-bit atomic world. And it hasn't had an update since then. Microsoft really should have kept investing in C++Amp IMO. ------------- ROCm is fine if you get the hardware and if it remains supported. But I think in practice, people expect support longer than what AMD is willing to give.

Const-me 3y ago
> 32-bit only atomics, I don't think its a very serious solution in practice Yeah, I think I encountered that while porting a hash map from CUDA to HLSL. However, I'm not sure that's necessarily a huge deal. Probably not an issue for machine learning or BLAS stuff, these use cases don't need fine-grained thread synchronization. For applications which would benefit from such synchronization, traditional lock-free techniques ported from CPU (i.e. compare and swap atomics on global memory) can be slow due to huge count of active threads on GPUs. I mean it's obviously better than locks, but sometimes it's possible to do something better instead of CAS. > so much API-crap you need to even get Hello World / SAXY up. I agree for D3D12 and especially Vulkan, but IMO D3D11 is not terribly bad. It has a steep learning curve, but the amount of boilerplate for simple apps is IMO reasonable. Especially for ML or similar GPGPU stuff which only needs a small subset of the API: compute shaders only, no textures, no render targets, no depth-stencil views, no input layouts, etc. However, unlike simple apps, real-life ones often need profiler and queue depth limiter, relatively hard to implement on top of these queries. I think Microsoft should ship them both in Windows SDK.
- dragontamer 3y ago