5 ms·

Makes me consider writing a post on misconceptions of GPU computing, such as requiring the problem to be fully data-parallel.

etdeagle2 2y ago
Please do! I would love to read about it. I have been playing with GPU hash tables shared between all the threads using locking (Interlocked.CompareExchange) and such in a Compute Shader. I have been wondering if there are better ways than locking.
- jms55 2y ago
  Hash tables on GPUs are cool. You can use them for meshless radiance caches, that automatically (for better or for worse) adapt to the surrounding geometry.
- winwang 2y ago
  Until I get around to it, there's this blog post I wrote: https://wiwa.substack.com/p/can-we-10x-rust-hashmap-throughput https://wiwa.substack.com/p/can-we-10x-rust-hashmap-throughp... Not an "expert" on GPU hash tables, but I'm mildly aware of other hashmap things like a novel(?) hash-indexed sorted array (HISA): https://arxiv.org/html/2311.02206v3 https://arxiv.org/html/2311.02206v3
jms55 2y ago
In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data transfer is a _really_ overlooked issue. People think "oh this is a parallel problem, I can have the GPU do it" and completely discount the cost to send the data to the GPU, and then get it back. If you want to write 20mb of data to a buffer, that's not just a memcpy, all that data has to go over the PCIe buss to the GPU (which again, is a completely separate device unless you're using an iGPU), and that's going to be expensive (in real time contexts). Similarly if you want to read a whole large buffer of results back from the GPU, that's going to take some time.
- physicsguy 2y ago
  I found that, I recently worked on a project where there was a Python library using pymc3, scikit-learn and pytensor that is used for time sensitive calculations. The performance wasn't very good and someone was pushing for us to spend a lot of time moving it to GPU, but as I had to point out, that means almost a total rewrite, not simply switching to the GPU enabled version(s) of those libraries, where they even exist.
- wiz21c 2y ago
  Another one I had when starting was to underestimate the power of the GPU. I really needed to increase the size of my toy problems to actually see the benefits of using a GPU. For data transfers, my experience is that you rather quickly hit that bottleneck, and it's a tough one. And it's not proportional to the number of transferred bits: transferring one byte naively can be extremely costly (like half the performance, I'm not joking)
- ElFitz 2y ago
  > Another kind of misconception: data transfer is a _really_ overlooked issue. […] If you want to write 20mb of data to a buffer, that's not just a memcpy, all that data has to go over the PCIe buss to the GPU […], and that's going to be expensive (in real time contexts). Similarly if you want to read a whole large buffer of results back from the GPU, that's going to take some time. Does having a unified memory, like Apple’s M-series chips, help with that?
  - b3orn 2y ago
    In theory yes, because you wouldn't need to copy the data, in practice it depends on the API and you might end up copying data from RAM to RAM. If the API doesn't allow you to simply pass an address to the GPU then you need to allocate memory on the GPU and copy your data to that memory, even if it's unified memory.
    - Kon-Peki 2y ago
      For Apple specifically, you have to act as if you do not have unified memory because Apple still supports discrete GPUs in Metal and also Swift is reference counted - the CPU portion of the app has no idea if the GPU portion is still using something (remember that the CPU and GPU are logically different devices even when they are on the same die). When you are running your code on an M- or A-series processor, most of that stuff probably ends up as no-ops. But worse case is that you copy from RAM to RAM, which is extraordinarily faster than pushing anything across the PCIe bus.
forgotpasagain 2y ago
Yes! I always thought shared memory is bigger and registers+l1s are per core instead of per SM.

etdeagle2 2y ago

Please do! I would love to read about it. I have been playing with GPU hash tables shared between all the threads using locking (Interlocked.CompareExchange) and such in a Compute Shader. I have been wondering if there are better ways than locking.

jms55 2y ago
Hash tables on GPUs are cool. You can use them for meshless radiance caches, that automatically (for better or for worse) adapt to the surrounding geometry.
winwang 2y ago
Until I get around to it, there's this blog post I wrote: https://wiwa.substack.com/p/can-we-10x-rust-hashmap-throughput https://wiwa.substack.com/p/can-we-10x-rust-hashmap-throughp... Not an "expert" on GPU hash tables, but I'm mildly aware of other hashmap things like a novel(?) hash-indexed sorted array (HISA): https://arxiv.org/html/2311.02206v3 https://arxiv.org/html/2311.02206v3

jms55 2y ago

In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc. You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet. Another kind of misconception: data transfer is a _really_ overlooked issue. People think "oh this is a parallel problem, I can have the GPU do it" and completely discount the cost to send the data to the GPU, and then get it back. If you want to write 20mb of data to a buffer, that's not just a memcpy, all that data has to go over the PCIe buss to the GPU (which again, is a completely separate device unless you're using an iGPU), and that's going to be expensive (in real time contexts). Similarly if you want to read a whole large buffer of results back from the GPU, that's going to take some time.

physicsguy 2y ago
I found that, I recently worked on a project where there was a Python library using pymc3, scikit-learn and pytensor that is used for time sensitive calculations. The performance wasn't very good and someone was pushing for us to spend a lot of time moving it to GPU, but as I had to point out, that means almost a total rewrite, not simply switching to the GPU enabled version(s) of those libraries, where they even exist.
wiz21c 2y ago
Another one I had when starting was to underestimate the power of the GPU. I really needed to increase the size of my toy problems to actually see the benefits of using a GPU. For data transfers, my experience is that you rather quickly hit that bottleneck, and it's a tough one. And it's not proportional to the number of transferred bits: transferring one byte naively can be extremely costly (like half the performance, I'm not joking)
ElFitz 2y ago
> Another kind of misconception: data transfer is a _really_ overlooked issue. […] If you want to write 20mb of data to a buffer, that's not just a memcpy, all that data has to go over the PCIe buss to the GPU […], and that's going to be expensive (in real time contexts). Similarly if you want to read a whole large buffer of results back from the GPU, that's going to take some time. Does having a unified memory, like Apple’s M-series chips, help with that?
- b3orn 2y ago
  In theory yes, because you wouldn't need to copy the data, in practice it depends on the API and you might end up copying data from RAM to RAM. If the API doesn't allow you to simply pass an address to the GPU then you need to allocate memory on the GPU and copy your data to that memory, even if it's unified memory.
  - Kon-Peki 2y ago
    For Apple specifically, you have to act as if you do not have unified memory because Apple still supports discrete GPUs in Metal and also Swift is reference counted - the CPU portion of the app has no idea if the GPU portion is still using something (remember that the CPU and GPU are logically different devices even when they are on the same die). When you are running your code on an M- or A-series processor, most of that stuff probably ends up as no-ops. But worse case is that you copy from RAM to RAM, which is extraordinarily faster than pushing anything across the PCIe bus.

forgotpasagain 2y ago

Yes! I always thought shared memory is bigger and registers+l1s are per core instead of per SM.