3 ms·

Can someone explain like I'm javascript, what's the deal with GPGPU? My naive understanding is that a graphics card is just a funny computer on which you can u

by wruza 3y ago

Can someone explain like I'm javascript, what's the deal with GPGPU? My naive understanding is that a graphics card is just a funny computer on which you can upload opcodes and data and let it cook itself. Why is CUDA such a big deal? Can't AMD just give direct access to its GPU as if it was an array of 4096 Arduino boards?

dist-epoch 3y ago
CUDA is like the TypeScript compiler which takes your nice code and turns it into something the browser (NVIDIA GPU) can run. AMD only has CoffeScript and it sucks compared to the TypeScript from NVIDIA.
UncleEntity 3y ago
[flagged]
- deleted 3y ago
  [deleted]
dotnet00 3y ago
The kind of GPGPU code where the language is a thin architecture agnostic layer over the opcodes is how most GPU code (eg shaders used in graphics applications) are implemented. This is also how OpenCL, Vulkan Compute etc do their thing. This approach requires a lot of boilerplate and babysitting, but works well for relatively short bits of code. CUDA is much higher level. It's roughly on par with a "C++ is C with classes" level in terms of language capability. This makes it much easier to develop complex applications. The C compatibility means that you can reuse the exact same code between CPU and GPU in many cases. It eliminates a lot of boilerplate, since you don't need to manage your data in as much detail (eg, while you still have to make sure your pointers are valid for the GPU, the code for uploading the function arguments is generated by the CUDA compiler). The value add that makes CUDA especially strong is all the first party libraries which have been carefully optimized and have widespread and proven long term support.
- wruza 3y ago
  Ah, I see. Thanks to everyone in this subthread for explanations!
- mFixman 3y ago
  Why is it so hard to develop a CUDA-like to AMD-opcode compiler? If it wasn't taking the tech community so long I would imagine it would not be harder than porting GCC to a new architecture.
  - dotnet00 3y ago
    I think the issues are mostly relating to getting the optimizer right. After all, not much of a point to GPU acceleration if it isn't meaningfully faster than CPU. A lot of these compatibility layers have this issue, DirectML, ZLUDA, etc. GPUs tend to expect the compiler to bake in things like instruction reordering/out of order execution for optimal performance. The other challenge is that there isn't an "AMD-opcode", each generation tends to change around the opcodes a bit, so you want to compile to an intermediate representation which the driver would ingest and compile down to what the GPU uses. NVIDIA uses PTX for this, it works very well. AMD's ROCm doesn't use an IR, it compiles the code for each architecture, which means they have a very limited support window for consumer GPUs (to limit binary size). OpenCL and Vulkan supports SPIR-V as an IR, but IIRC, the OpenCL SPIR-V on AMD is very buggy, and Vulkan SPIR-V is very different. This has other side effects, like BLAS library support range. Hard for an open source community to justify putting in tons of effort into optimizing an entire BLAS library for each generation, when it'll only be supported for ~4 years (so, by the time you're finishing up with driver stability and library optimization, you're probably already at least a quarter of the way through the support period).
    - mFixman 3y ago
      Neat, that makes sense. Still, giant missed opportunity for AMD not to focus on this when NVIDIA has an almost monopoly on massive GPGPUs.
mnau 3y ago
Cuda is big deal because it works on every nv hw. Plus theyvhave fine tuned sw that utiluzes hw to max. Amd doesn't. Go watch https://www.youtube.com/watch?v=NPinFkavsrk https://www.youtube.com/watch?v=NPinFkavsrk or https://www.youtube.com/watch?v=AqPIOtUkxNo https://www.youtube.com/watch?v=AqPIOtUkxNo That is an attempt to avoid buggy amd implementation and go closer to hw. Everything crashes (kernel), their own demos lock up the card and so on.
- wruza 3y ago
  At first I thought "not gonna watch a 6+ hour stream", but decided to give it a go anyway. Those who are interested and have at least ~some assembly background may find all the funny things in the first 10 minutes of the first video, and that isn't even a highlight compilation. I understand the problem much deeper now :) thanks!
  - mnau 3y ago
    Yea, in the 5:50 of video: Oh no, AMD driver just dereferenced null pointer. It has been a while since I watched these streams, but that is about the theme of all six hours (and few others streams). It's just complete mine field, where he is trying to walk through a very narrow path of success. Combine it with things like this (they now actually have a improved documentation, significant progress): https://github.com/ROCm/ROCm/issues/1714 https://github.com/ROCm/ROCm/issues/1714 or this (HIP doesn't support L2 cache coherency, so disable GPU L2 cache) https://docs.amd.com/projects/HIP/en/latest/user_guide/hip_porting_guide.html#threadfence-system https://docs.amd.com/projects/HIP/en/latest/user_guide/hip_p... and it's just FUBAR.
    - zero_k 3y ago
      Haha null deref is funny. Not surprised they are so bad still. They were terrible back then. Didn't improve over time.

by wruza 3y ago

dist-epoch 3y ago

CUDA is like the TypeScript compiler which takes your nice code and turns it into something the browser (NVIDIA GPU) can run. AMD only has CoffeScript and it sucks compared to the TypeScript from NVIDIA.

UncleEntity 3y ago

[flagged]

deleted 3y ago
[deleted]

dotnet00 3y ago

The kind of GPGPU code where the language is a thin architecture agnostic layer over the opcodes is how most GPU code (eg shaders used in graphics applications) are implemented. This is also how OpenCL, Vulkan Compute etc do their thing. This approach requires a lot of boilerplate and babysitting, but works well for relatively short bits of code. CUDA is much higher level. It's roughly on par with a "C++ is C with classes" level in terms of language capability. This makes it much easier to develop complex applications. The C compatibility means that you can reuse the exact same code between CPU and GPU in many cases. It eliminates a lot of boilerplate, since you don't need to manage your data in as much detail (eg, while you still have to make sure your pointers are valid for the GPU, the code for uploading the function arguments is generated by the CUDA compiler). The value add that makes CUDA especially strong is all the first party libraries which have been carefully optimized and have widespread and proven long term support.

wruza 3y ago
Ah, I see. Thanks to everyone in this subthread for explanations!
mFixman 3y ago
Why is it so hard to develop a CUDA-like to AMD-opcode compiler? If it wasn't taking the tech community so long I would imagine it would not be harder than porting GCC to a new architecture.
- dotnet00 3y ago
  I think the issues are mostly relating to getting the optimizer right. After all, not much of a point to GPU acceleration if it isn't meaningfully faster than CPU. A lot of these compatibility layers have this issue, DirectML, ZLUDA, etc. GPUs tend to expect the compiler to bake in things like instruction reordering/out of order execution for optimal performance. The other challenge is that there isn't an "AMD-opcode", each generation tends to change around the opcodes a bit, so you want to compile to an intermediate representation which the driver would ingest and compile down to what the GPU uses. NVIDIA uses PTX for this, it works very well. AMD's ROCm doesn't use an IR, it compiles the code for each architecture, which means they have a very limited support window for consumer GPUs (to limit binary size). OpenCL and Vulkan supports SPIR-V as an IR, but IIRC, the OpenCL SPIR-V on AMD is very buggy, and Vulkan SPIR-V is very different. This has other side effects, like BLAS library support range. Hard for an open source community to justify putting in tons of effort into optimizing an entire BLAS library for each generation, when it'll only be supported for ~4 years (so, by the time you're finishing up with driver stability and library optimization, you're probably already at least a quarter of the way through the support period).
  - mFixman 3y ago
    Neat, that makes sense. Still, giant missed opportunity for AMD not to focus on this when NVIDIA has an almost monopoly on massive GPGPUs.

mnau 3y ago

Cuda is big deal because it works on every nv hw. Plus theyvhave fine tuned sw that utiluzes hw to max. Amd doesn't. Go watch https://www.youtube.com/watch?v=NPinFkavsrk https://www.youtube.com/watch?v=NPinFkavsrk or https://www.youtube.com/watch?v=AqPIOtUkxNo https://www.youtube.com/watch?v=AqPIOtUkxNo That is an attempt to avoid buggy amd implementation and go closer to hw. Everything crashes (kernel), their own demos lock up the card and so on.

wruza 3y ago
At first I thought "not gonna watch a 6+ hour stream", but decided to give it a go anyway. Those who are interested and have at least ~some assembly background may find all the funny things in the first 10 minutes of the first video, and that isn't even a highlight compilation. I understand the problem much deeper now :) thanks!
- mnau 3y ago
  Yea, in the 5:50 of video: Oh no, AMD driver just dereferenced null pointer. It has been a while since I watched these streams, but that is about the theme of all six hours (and few others streams). It's just complete mine field, where he is trying to walk through a very narrow path of success. Combine it with things like this (they now actually have a improved documentation, significant progress): https://github.com/ROCm/ROCm/issues/1714 https://github.com/ROCm/ROCm/issues/1714 or this (HIP doesn't support L2 cache coherency, so disable GPU L2 cache) https://docs.amd.com/projects/HIP/en/latest/user_guide/hip_porting_guide.html#threadfence-system https://docs.amd.com/projects/HIP/en/latest/user_guide/hip_p... and it's just FUBAR.
  - zero_k 3y ago
    Haha null deref is funny. Not surprised they are so bad still. They were terrible back then. Didn't improve over time.