9 ms·

GPU Programming: When, Why and How?

165 points by transpute 3y ago

kaycey2022 3y ago
No local setup it seems :(
whatever1 3y ago
Hard No for sequential / dynamic programming code and sparse matrices.
- patagurbon 3y ago
  Sparse matrix codes can be and often are much faster than their CPU counterparts. At least for the most common kernels
  - adrian_b 3y ago
    Moreover, NVIDIA provides the free cuSPARSE library, which can be good enough to accelerate many common operations with sparse matrices. Only when that does not already cover all your needs you have to find or develop other software for handling sparse matrices on GPUs.
  - syockit 3y ago
    My experience with it was: with CSR format, SpMV got about 40x speedup compared to a single CPU core on a RTX 3090, with full core utilization. Rough calculation tells me it's equivalent to 80 GFlops, which is far from the full potential of the card. I thought there was someting wrong with cuSPARSE so I went ahead and wrote a SpMV kernel, based on the code in NVIDIA's past reports. It actually performed slightly better. Fair enough, cuSPARSE has to support all kind of workloads so it might be just that in my case, the handwritten kernel worked better. But still, it would never get to the regime of hundreds of GFlops. After reading various literatures on GPU SpMV implementations getting to the same conclusion, I had to concede that I can't squeeze more GFlops out of it.
    - KeplerBoy 3y ago
      Have you checked the roofline performance model as reported by Nsight Compute? A lot of workloads can never reach the hundreds of GFlops because they are bandwidth limited.
      - syockit 3y ago
        I couldn't check it using Nsight Compute. It didn't find the kernel even though I ran it as root. A40 is supposed to be Ampere architecture, with GA102 chip so it should be supported. Maybe the toolkit I'm using is too old (CUDA 11.3). Everything is on the device during the tight loop, and there is minimal transfer to host, so I don't think it's bandwidth limited in that sense. It could be bandwidth limited on the cache level but I couldn't tell. I wish I could get Compute running.
hutzlibu 3y ago
Interesting, but very heavy read. And this is about the low level access towards it via system libaries. A more simple way to get into GPU programming, would be WebGPU. I can recommend this tutorial as an introduction: https://surma.dev/things/webgpu/ https://surma.dev/things/webgpu/ GPU programming is really a different way of programming, even though the code looks mostly familiar. And if you are used to single threaded, it will give you a massive headache thinking about, how to do everything in parallel, so that everything is not blocking everything. The raw power is tempting, so much that I evaluate now how to do allmost everything on the GPU, but many things simply are not suitable to do there. Basically, you want to avoid conditions and loops. Cannot do recursion. Debugging is a pain and non obvious things can make everything very slow, until you roughly understand, what is going on inside.
- onion2k 3y ago
  how to do everything in parallel I found it was worse than that when I started playing with shaders, because my idea of parallelism came from threads, workers, and async functions eg doing several separate tasks at the same time. Parallel on a GPU is more like running lots of the same function each with different parameters at the same time eg running a function to calculate the color of a pixel using its x and y coordinates (and maybe some other uniforms) for every pixel on the screen.
  - delta_p_delta_x 3y ago
    I like to put it like this: a GPU is a really, really fast Map-Reduce hardware processor. You give it a bunch of data with elements which are all independent of each other, apply the same transformations to to each element, and output a single data structure (a colour attachment or render target, or some compute buffer). This explains a lot of the shortcomings and design decisions of GPUs. They prioritise throughput over latency; they have a lot of memory but much slower than the cores themselves (~500 – 700 cycle latency compared to ~20 – 40 cycle latency on the CPU); they run 'threads' in synchronised lock-step, called warps; branching on a GPU is particularly expensive—if one shader for one fragment needs to branch, then the entire warp that it's on will have to wait for it, or worse still, calculate that branch, too; Their 3D graphics legacy still endures: despite the advent of GPGPU, they still have plenty of dedicated hardware for the graphics pipeline, including rasterisers, texture-mapping units, render output processors, etc. Once people get this idea, then GPU programming is easier to reason about. One execution of a shader deals with one fragment (actually four, but at a high level of abstraction, this is sufficient). Oddly enough, I did GPU parallelisation first, and then came to CPU parallelisation. I find CPU-level parallelism and concurrency problems like OS inter-process synchronisation, async-await, etc. much harder to reason about, because they generally cannot be reduced to such a simplistic 'one thread for one pixel'. It's also why I like embarrassingly parallel problems, because they're the most easily GPU-able.
wiz21c 3y ago
I still don't get the "no good for branching code" point... Since most of the work is split in millions of "threads" which all run in parallel, then why is branching an issue ? Some threads will take a bit longer than the others. Maybe groups of threads will go as slow as the slowest but still, prallelism should make up the bulk of the acceleration. No ?
- maeln 3y ago
  Most GPU's only have so much compute cores. Even if you have a million threads, you might only have a few hundreds cores to execute them. If you have heavy branching you might slow down the whole lane (which size can vary from 8 to 64 execution most of the time). Which at this scale still do make a big difference. Although for small branching, masking help avoid too much slowdown.
  - WJW 3y ago
    To expand on this, even the latest Nvidia 4090 card "only" has about 16k cores. That's a lot, but it's hardly "millions" either.
    - ben-schaaf 3y ago
      To expand further the 16k cores aren't real cores. What people would normally consider a core is what Nvidia calls a "streaming multiprocessor" (SM), of which the 4090 has 128. These cores each have 128 "cuda cores", analogous to a SIMD lane - although not quite since "cuda cores" are themselves SIMD. To simplify: Each 128 SMs can run a different program and each of their 128 "cuda cores" can do ~single-cycle (small) matrix operations.
      - 6keZbCECT2uB 3y ago
        I'm not sure about the 4090, but most of the GPUs I use have a warp size of 32, and warp divergence affects only up to those 32 threads. If you have a branch and all threads agree, you only walk down one branch. My mental model is a bit more like you have collections of warps in a block, and all warps in a block get scheduled onto an SM. Different GPU architectures allow for different numbers of warps to be simultaneously active or inactive, and each warp has its own instruction pointer and can be suspended while waiting for things like memory. I found the picture on pg 22 here really helpful: https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf https://images.nvidia.com/aem-dam/en-zz/Solutions/data-cente... Note that although there's 4 schedulers, on the A100, they don't dispatch every cycle iirc.
Munksgaard 3y ago
Shameless plug for Futhark[0], which you can use to write your computational kernels. The idea is to use functional constructs like map and reduce to express parallel code and let the compiler handle the generation of low-level OpenCL or CUDA. Disclaimer: I've recently finished a PhD focused on memory optimizations in Futhark. [0]: https://futhark-lang.org/ https://futhark-lang.org/ Edit: They do actually mention stuff like Julia and NumPy.
- sspiff 3y ago
  I used Futhark in the past. It's probably been 5 years or more, so a lot may have changed since then. It's absolutely a much lower barrier to entry, especially if you are familiar with functional programming. However, it also leaves a lot of performance on the table, and I found it to be hard or cumbersome to integrate with languages other than Python at the time. And you're still stuck with CUDA (which is single vendor) or OpenCL (which is a pain to set up on most consumer systems, and has very lackluster driver quality for a lot of vendors). I would have liked that by now support for Vulkan compute or OpenGL compute shaders would have been added, but alas.
  - Munksgaard 3y ago
    Great to hear that you liked the language! Regarding the performance, I'd be interested to hear more about your experiences. Because we compile to CUDA or OpenCL in the end, we cannot claim to be faster than what you could (in principle) write in hand. However, most of our benchmarks compare favorably with handwritten reference implementations, and the heavily optimizing compiler is able to write code that is tough to write by hand. However, we're always looking for instances where we can do better. The goal is to be comparable to CUDA/OpenCL in as many cases as possible.
    - sspiff 3y ago
      I'd love to tell you what it did suboptimally in my usecase at the time, but I no longer have the code and it's been so long I don't remember the specifics. Like I said in my earlier comment, it was so long ago the performance remarks may be outdated and the project may have resolved them. I would like to give Futhark another try, but I need easy and portable integration in either Go or Rust, and as far as I can tell, that's not the case today? For my current project, I ended up selecting webgpu for my GPU compute needs.
isentrop1c 3y ago
GPU is one part where modern design of Julia truly shines as it allows seamless reusability and composability. I wish the language gets the proper recognition and adoption
EVa5I7bHFq9mnYK 3y ago
I used ILGPU C# library, where I just wrote my kernel as a regular C# function, and the library executed it on the GPU in parallel. Don't need to know nothin' about blocks, warps and them other scary stuff. All thanks to C# excellent reflection support. Learned that computation should be highly local (working on small isolated pieces of data) and without if's, otherwise won't get any performance benefits.

165 points by transpute 3y ago

kaycey2022 3y ago

No local setup it seems :(

whatever1 3y ago

Hard No for sequential / dynamic programming code and sparse matrices.

patagurbon 3y ago
Sparse matrix codes can be and often are much faster than their CPU counterparts. At least for the most common kernels
- adrian_b 3y ago
  Moreover, NVIDIA provides the free cuSPARSE library, which can be good enough to accelerate many common operations with sparse matrices. Only when that does not already cover all your needs you have to find or develop other software for handling sparse matrices on GPUs.
- syockit 3y ago
  My experience with it was: with CSR format, SpMV got about 40x speedup compared to a single CPU core on a RTX 3090, with full core utilization. Rough calculation tells me it's equivalent to 80 GFlops, which is far from the full potential of the card. I thought there was someting wrong with cuSPARSE so I went ahead and wrote a SpMV kernel, based on the code in NVIDIA's past reports. It actually performed slightly better. Fair enough, cuSPARSE has to support all kind of workloads so it might be just that in my case, the handwritten kernel worked better. But still, it would never get to the regime of hundreds of GFlops. After reading various literatures on GPU SpMV implementations getting to the same conclusion, I had to concede that I can't squeeze more GFlops out of it.
  - KeplerBoy 3y ago
    Have you checked the roofline performance model as reported by Nsight Compute? A lot of workloads can never reach the hundreds of GFlops because they are bandwidth limited.
    - syockit 3y ago
      I couldn't check it using Nsight Compute. It didn't find the kernel even though I ran it as root. A40 is supposed to be Ampere architecture, with GA102 chip so it should be supported. Maybe the toolkit I'm using is too old (CUDA 11.3). Everything is on the device during the tight loop, and there is minimal transfer to host, so I don't think it's bandwidth limited in that sense. It could be bandwidth limited on the cache level but I couldn't tell. I wish I could get Compute running.

hutzlibu 3y ago

Interesting, but very heavy read. And this is about the low level access towards it via system libaries. A more simple way to get into GPU programming, would be WebGPU. I can recommend this tutorial as an introduction: https://surma.dev/things/webgpu/ https://surma.dev/things/webgpu/ GPU programming is really a different way of programming, even though the code looks mostly familiar. And if you are used to single threaded, it will give you a massive headache thinking about, how to do everything in parallel, so that everything is not blocking everything. The raw power is tempting, so much that I evaluate now how to do allmost everything on the GPU, but many things simply are not suitable to do there. Basically, you want to avoid conditions and loops. Cannot do recursion. Debugging is a pain and non obvious things can make everything very slow, until you roughly understand, what is going on inside.

onion2k 3y ago
how to do everything in parallel I found it was worse than that when I started playing with shaders, because my idea of parallelism came from threads, workers, and async functions eg doing several separate tasks at the same time. Parallel on a GPU is more like running lots of the same function each with different parameters at the same time eg running a function to calculate the color of a pixel using its x and y coordinates (and maybe some other uniforms) for every pixel on the screen.
- delta_p_delta_x 3y ago
  I like to put it like this: a GPU is a really, really fast Map-Reduce hardware processor. You give it a bunch of data with elements which are all independent of each other, apply the same transformations to to each element, and output a single data structure (a colour attachment or render target, or some compute buffer). This explains a lot of the shortcomings and design decisions of GPUs. They prioritise throughput over latency; they have a lot of memory but much slower than the cores themselves (~500 – 700 cycle latency compared to ~20 – 40 cycle latency on the CPU); they run 'threads' in synchronised lock-step, called warps; branching on a GPU is particularly expensive—if one shader for one fragment needs to branch, then the entire warp that it's on will have to wait for it, or worse still, calculate that branch, too; Their 3D graphics legacy still endures: despite the advent of GPGPU, they still have plenty of dedicated hardware for the graphics pipeline, including rasterisers, texture-mapping units, render output processors, etc. Once people get this idea, then GPU programming is easier to reason about. One execution of a shader deals with one fragment (actually four, but at a high level of abstraction, this is sufficient). Oddly enough, I did GPU parallelisation first, and then came to CPU parallelisation. I find CPU-level parallelism and concurrency problems like OS inter-process synchronisation, async-await, etc. much harder to reason about, because they generally cannot be reduced to such a simplistic 'one thread for one pixel'. It's also why I like embarrassingly parallel problems, because they're the most easily GPU-able.

wiz21c 3y ago

I still don't get the "no good for branching code" point... Since most of the work is split in millions of "threads" which all run in parallel, then why is branching an issue ? Some threads will take a bit longer than the others. Maybe groups of threads will go as slow as the slowest but still, prallelism should make up the bulk of the acceleration. No ?

maeln 3y ago
Most GPU's only have so much compute cores. Even if you have a million threads, you might only have a few hundreds cores to execute them. If you have heavy branching you might slow down the whole lane (which size can vary from 8 to 64 execution most of the time). Which at this scale still do make a big difference. Although for small branching, masking help avoid too much slowdown.
- WJW 3y ago
  To expand on this, even the latest Nvidia 4090 card "only" has about 16k cores. That's a lot, but it's hardly "millions" either.
  - ben-schaaf 3y ago
    To expand further the 16k cores aren't real cores. What people would normally consider a core is what Nvidia calls a "streaming multiprocessor" (SM), of which the 4090 has 128. These cores each have 128 "cuda cores", analogous to a SIMD lane - although not quite since "cuda cores" are themselves SIMD. To simplify: Each 128 SMs can run a different program and each of their 128 "cuda cores" can do ~single-cycle (small) matrix operations.
    - 6keZbCECT2uB 3y ago
      I'm not sure about the 4090, but most of the GPUs I use have a warp size of 32, and warp divergence affects only up to those 32 threads. If you have a branch and all threads agree, you only walk down one branch. My mental model is a bit more like you have collections of warps in a block, and all warps in a block get scheduled onto an SM. Different GPU architectures allow for different numbers of warps to be simultaneously active or inactive, and each warp has its own instruction pointer and can be suspended while waiting for things like memory. I found the picture on pg 22 here really helpful: https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf https://images.nvidia.com/aem-dam/en-zz/Solutions/data-cente... Note that although there's 4 schedulers, on the A100, they don't dispatch every cycle iirc.

Munksgaard 3y ago

Shameless plug for Futhark[0], which you can use to write your computational kernels. The idea is to use functional constructs like map and reduce to express parallel code and let the compiler handle the generation of low-level OpenCL or CUDA. Disclaimer: I've recently finished a PhD focused on memory optimizations in Futhark. [0]: https://futhark-lang.org/ https://futhark-lang.org/ Edit: They do actually mention stuff like Julia and NumPy.

sspiff 3y ago
I used Futhark in the past. It's probably been 5 years or more, so a lot may have changed since then. It's absolutely a much lower barrier to entry, especially if you are familiar with functional programming. However, it also leaves a lot of performance on the table, and I found it to be hard or cumbersome to integrate with languages other than Python at the time. And you're still stuck with CUDA (which is single vendor) or OpenCL (which is a pain to set up on most consumer systems, and has very lackluster driver quality for a lot of vendors). I would have liked that by now support for Vulkan compute or OpenGL compute shaders would have been added, but alas.
- Munksgaard 3y ago
  Great to hear that you liked the language! Regarding the performance, I'd be interested to hear more about your experiences. Because we compile to CUDA or OpenCL in the end, we cannot claim to be faster than what you could (in principle) write in hand. However, most of our benchmarks compare favorably with handwritten reference implementations, and the heavily optimizing compiler is able to write code that is tough to write by hand. However, we're always looking for instances where we can do better. The goal is to be comparable to CUDA/OpenCL in as many cases as possible.
  - sspiff 3y ago
    I'd love to tell you what it did suboptimally in my usecase at the time, but I no longer have the code and it's been so long I don't remember the specifics. Like I said in my earlier comment, it was so long ago the performance remarks may be outdated and the project may have resolved them. I would like to give Futhark another try, but I need easy and portable integration in either Go or Rust, and as far as I can tell, that's not the case today? For my current project, I ended up selecting webgpu for my GPU compute needs.

isentrop1c 3y ago

GPU is one part where modern design of Julia truly shines as it allows seamless reusability and composability. I wish the language gets the proper recognition and adoption

EVa5I7bHFq9mnYK 3y ago

I used ILGPU C# library, where I just wrote my kernel as a regular C# function, and the library executed it on the GPU in parallel. Don't need to know nothin' about blocks, warps and them other scary stuff. All thanks to C# excellent reflection support. Learned that computation should be highly local (working on small isolated pieces of data) and without if's, otherwise won't get any performance benefits.