12 ms·

This is a talk I gave at Jane Street late February on GPU compute, especially using graphics API's such as Vulkan and the upcoming WGPU. Feel free to scan throu

This is a talk I gave at Jane Street late February on GPU compute, especially using graphics API's such as Vulkan and the upcoming WGPU. Feel free to scan through the slides [1], and to ask me anything. [1]: https://docs.google.com/presentation/d/1FRH81IW9RffkJjm6ILFZ7raCgFAUPXYYFXfiyKmhkx8/edit?usp=sharing https://docs.google.com/presentation/d/1FRH81IW9RffkJjm6ILFZ...

Jhsto 7y ago
Thanks for the video and slides, super interesting! One question: in one of the slides, you mention the Vulkan 1.2 / SPIR-V 1.5 Vulkan Memory model. What exactly is this and why do you think this is meaningful? Would you have any links to learn more? When I read the SPIR-V specification, it mentions that this kind of memory model exists, but it does not put any effort into explaining why it's worthwhile or how it works on a high level.
- jabl 7y ago
  Perhaps in a few years when we'll start to see things like CXL providing cache-coherency between main memory and GPU memory this will become important?
- raphlinus 7y ago
  That could be the subject of a whole other talk :) The memory model is needed to do more sophisticated communication between the parallel workgroups. The classic example is to make a queue, even a simple one such as a simple single-producer single-consumer ring buffer (for communication between a pair of workgroups). The way this works, the producer fills an element in the buffer, then bumps an index. The consumer observes the index, then reads the element from the buffer. Without a memory model, there's no way to guarantee that this ordering is preserved, and (to map this to hardware), the write of the index bump could flow through the caches, while the write of the element could still be "dirty" in the cache on the producer, so the consumer reads stale data. What the memory model does is provide the programmer an explicit way to express ordering constraints. So the write of the index bump is with "release" semantics and the read of that by the consumer is with "acquire" semantics, which guarantees that producer writes before the release are visible to consumer reads after the acquire. This unlocks a whole range of sophisticated concurrent data structures, such as concurrent hash maps, fancier memory allocation (even without a memory model, a simple allocate-only bump allocator is feasible), and so on. Thanks for the question and kind words, I hope this illuminates.
  - Jhsto 7y ago
    Thanks, the answer with some examples was exactly what I was hoping to get answered! Thank you for taking the time to respond.
throwlaplace 7y ago
of the links you put on the last page (or other gpu languages you might not have included) what's the most mature way to write decent gpu code right now (even if not necessarily optimal). i'm right this moment working on a project where i'm trying to speed up a blob detection algorithm by porting it to gpu. i'm using pytorch because the primitives fit naturally but if that weren't the case i think i would be kind of at a loss for how to do it (i do not know, as of yet, how to write cuda kernels). btw the Julia link is dead
- raphlinus 7y ago
  It depends on what you're trying to do. If you want to get stuff done now then just get an Nvidia card and write CUDA. The tools and knowledge community are vastly ahead of other approaches. If you're doing image-like stuff, Halide is good, there's lots of stuff shipping on it. One other project I'd add is emu, which is built on wgpu. I think that's closest to the future I have in mind, but still in early stages. https://github.com/calebwin/emu https://github.com/calebwin/emu
  - throwlaplace 7y ago
    okay in that case can you recommend a good tut for writing cuda code?
    - alexhutcheson 7y ago
      The official docs are pretty good. This is probably a good starting point: https://devblogs.nvidia.com/even-easier-introduction-cuda/ https://devblogs.nvidia.com/even-easier-introduction-cuda/ Nvidia also sponsored a Udacity course, but I don't have any direct experience with it: https://developer.nvidia.com/udacity-cs344-intro-parallel-programming https://developer.nvidia.com/udacity-cs344-intro-parallel-pr...
- ZeroCool2u 7y ago
  If you're interested in getting started writing GPU code and you're currently dealing with PyTorch a good place to start might be using NUMBA for CUDA programming. It lets you do in Python what you'd normally do in C/C++. It's great, because setup is minimal and it has some good tutorial materials. Plus, you're mostly emulating what you'd end up doing in C/C++, so conceptually the code is very similar. When I was at GTC 2018 this is one of the training sessions I went to and you can now access that here[1]. The official NUMBA for CUDA docs are great too[2]. Also, you can avoid a lot of installation/dependencies headaches by using anaconda/miniconda if you're doing this stuff locally. [1]: https://github.com/ContinuumIO/gtc2018-numba https://github.com/ContinuumIO/gtc2018-numba [2]:https://numba.pydata.org/numba-doc/latest/cuda/overview.html https://numba.pydata.org/numba-doc/latest/cuda/overview.html
sorenjan 7y ago
Do you think other alternatives to Cuda will have the same drawback as OpenCL, that Nvidia won't support it properly and therefore not be a viable option on their hardware which is dominating the market at the moment?
- gmueckl 7y ago
  Well, what are the chances that less vendor lock in is in nVidia's interest?
  - angry_octet 7y ago
    Is it vendor lock-in if the clients want to be there? It's more like you can be in the club with the good music and hip people, or the club with the average unknown DJ and an empty dance floor.
    - bl0b 7y ago
      That makes sense for GPU-compute-in-the-cloud applications where you can pick the hardware and don't care if other hardware doesn't work, but some people want to write programs that use GPGPU that can run on anyone's machine.
- raphlinus 7y ago
  No, precisely because I see the emerging alternatives as being based on Vulkan and SPIR-V, which I believe have momentum far greater than OpenCL. The only question is whether a compute platform based on these standards and open source tools could be anywhere nearly as good as CUDA. I believe we're far from that today, but it is possible and there's a lot of good work in that direction.
  - muro 7y ago
    Doesn't that exclude Apple?
    - 7y ago
sprash 7y ago
Halfway through you were bringing up hopes for a great new way to do 2D rendering on GPUs but then there is no show of actual implementation. Rather anticlimactic! Even decades old Cairo is beating VulKan/GPU based renderers like Skia in some tasks. Can it be that hardware intended for ML or 3D-graphics is not suited at all for 2D rendering (parallelism vs. pipelining)? How will WebGPU solve those problems? To me this talk raises more questions than it answers...
- raphlinus 7y ago
  It's work in progress. I was hoping to get more done on the renderer but various things have been slowing me down. But lately I've been feeling a new burst of energy, so hope to have something more to show before long. I agree that the talk raises a lot of questions. Hopefully that's a good thing, as I think they deserve the attention of curious people! Regarding the mismatch between 3D hardware and 2D rendering, we'll need hard empirical numbers, but the approach I'm taking is basically to write a software renderer (not unlike Cairo or even libart which predates it), just one that's designed for massive parallelism so it can run on the GPU. This will have performance characteristics more similar to Cairo (just faster as so much more throughput is available) than rasterization-based approaches. The latter really fall down when there's soft clipping and lots of blending, as that requires generation of intermediate texture buffers. I do address this in the talk :) I hope I will have more solid answers for some of these questions before long.
  - sprash 7y ago
    I'm experimenting with GPU based 2D rendering using signed distance functions as drawing primitives for quiet a long time now. My experience is that as soon as the object tree exceeds a certain size the CPU will always beat the GPU. The only way the GPU can win is when you have high complexity drawing primitives and a low amount of objects which is exactly the opposite one usually encounters in the real world. Maybe the partitioning approach you are proposing will help but only real world implementations and benchmarks will tell.
    - 7y ago
zozbot234 7y ago
Is Vulkan really becoming viable for compute? It seems like something that's obviously worth exploring given the overlap with more traditional graphics workloads, but I have seen some claims that it's still lagging behind OpenCL wrt. some features, and that it could be good for some things but is not obviously competitive across the board. It's possible that this impression is outdated though.
- raphlinus 7y ago
  I think the answer to this is emerging, but I would say yes. Vulkan and SPIR-V are rapidly gaining features, while OpenCL is stagnating. Some of these features are very recent, like pointers and a memory model. Another area I'm expecting Vulkan to pull ahead is support for efficient float16 computation, which is important for ML workloads, and is increasingly well supported in hardware. One thing to keep in mind though, programming directly in Vulkan is too low-level for most users, while OpenCL is a fairly friendly API. We need tools that can compile to SPIR-V and deploy on Vulkan, but high enough level you can express your problem rather than (for example) manually placing pipeline barriers between dispatches. The lack of such tools, and having to build some of my own, is one of the reasons for the slowdown in progress on my renderer project.
  - Jasper_ 7y ago
    CUDA is unfortunately very, very far ahead of Vulkan. "Graphics Compute" is quite different from CUDA at the advanced levels if you've never used it.
  - pjmlp 7y ago
    In CUDA I can just use ISO C++17 (C++20 around the corner) plus CUDA extensions, or any other language that targets PTX, with very nice graphical debugging from what is going on the GPGPU. OpenCL is playing catch up with CUDA, but it requires at least version 2.0 for similar capabilities, while many vendors are still shipping 1.x drivers. Vulkan, well it seems to be the same story as OpenCL regarding compute tooling, but lets see how much love it will get from NVidia. I advise watching the GTC 2020 talks regarding CUDA and Vulkan compute.
Symmetry 7y ago
Any opinion on ROCm?
- mpreda 7y ago
  ROCm provides a good OpenCL 2.x implementation targeting AMD GCN. The compiler/optimizer is clang-based. It is open-source. One great thing is that ROCm OpenCL supports GCN assembly in the OpenCL source, which is handy in some corner-cases that are not properly handled by the optimizer.
- VHRanger 7y ago
  Doesn't support many platforms
Strilanc 7y ago
What is a reasonable lower limit on latency that I could expect from a GPU pipeline? For example, I know that if I use WebGL and issue a `readPixels` call then that seems to take on the order of 1 to 10 milliseconds no matter what I do. This suggests that if I needed to react to something in under 100 microseconds, I probably shouldn't have the computation of the reaction involve WebGL. But is that true of GPU compute in general, or just an artifact of the abstraction exposed by WebGL?
- jdashg 7y ago
  There's a bunch of intractable latency, but there are patterns that can help maintain pipelining, if you don't need the data immediately: https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/WebGL_best_practices#Non-blocking_async_data_downloadreadback https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/W...
  - Strilanc 7y ago
    Right, but if I go closer to the metal how much of the latency is really truly intractable? 100 microseconds? 1 millisecond? 10 milliseconds?
    - bl0b 7y ago
      Can't speak for WebGL APIs, but in OpenGL (and other C++ graphics/GPGPU platforms), it is certainly possible (and necessary) to write fully interactive real-time apps that rely on a few rounds of communication between GPU and CPU code every frame. At 60 fps, you have 16 ms per frame. As long as you aren't copying huge arrays back and forth, I'd say you can budget in the 1 ms order-of-magnitude per data transfer. I don't know much of that would be avoidable if you were actually closer to the metal than the raw OpenGL C++ API call
- kllrnohj 7y ago
  You do have to design around the GPU typically being almost like a server. It's very fast, but also "very far" away - both in latency & in bandwidth. PCI-E 3.0 x16 is 'only' 16GB/s. That's fast for I/O, but still in the ballpark of being like I/O & not like RAM. And that's if you even get x16 electrical - x8 for GPUs is pretty common in eg. laptops where the other lanes go to things like thunderbolt 3. Integrated graphics are a different story, they often just have a direct connection to DRAM and CPU<->GPU communication is therefore super fast & low latency. In theory, anyway, if the API abstractions even let you share memory between the two. So that means even for things where GPUs should do really well at it, like summing two arrays together, doing it on the CPU can still be a lot faster if the inputs & outputs are local to the CPU.
2008guy 7y ago
What were you talking about with the socialist revolution?
- mkl 7y ago
  He was giving a talk to people from Jane Street [1], who make money on the stock market using OCaml (mentioned in the same bit). [1] https://en.wikipedia.org/wiki/Jane_Street_Capital https://en.wikipedia.org/wiki/Jane_Street_Capital
  - 2008guy 7y ago
    Right but he basically asserted to everyone that a socialist revolution is coming to wipe them out. Is there something I don’t know? Should I be getting ready for this?
    - mkl 7y ago
      Huh? It was a joke. Explaining the joke will kill it, but here goes: He was saying that their OCaml skills would still be useful even if they couldn't make money in finance because capitalism fell over. The audience got the joke and laughed. It's nothing like what you're saying.
      - 2008guy 7y ago
        What I saw was this guy flatly asserting that a socialist revolution is coming, I thought at first he was talking about Bernie Sanders, and where there should be a punchline there is no punchline. Then everyone gives an awkward and half-cocked chuckle as one does when presented with something so bizarre.
fluffything 7y ago
I write a lot of compute kernels in CUDA, and my litmus test is prefix sum, for two reasons. First, you can implement it in pure CUDA C++, and max out the memory bandwidth of any nvidia or AMD GPU. The CUB library provides a state of the art implementation (using decoupled-lookback) that one can compare against new programming languages. Second, it is one of the most basic parallel algorithm building blocks. Many parallel algorithms use it, and many parallel algorithms are "prefix-sum-like". If I am not able to write prefix sum from scratch efficiently in your programming language / library, I can't use it. Every time someone shows a new programming language for compute, the examples provided are super basic (e.g. `map(...).fold(...)`), but I have yet to see a new programming language that can be used to implement the 2-3 most fundamental parallel algorithms from any parallel algorithms graduate course. For example, Futhark provides a prefix sum intrinsic, that just calls CUB - if you want to implement a prefix-sum-like algorithm, you are out of luck. In WGPU, it appears that prefix-sum will be an intrinsic of WHSL, which sounds like you would be out-of-luck too. You mentioned WGPU and Vulkan. Do you know how to implement prefix-sum from scratch on these? If so, do you know how the performance compare against CUB?
- raphlinus 7y ago
  That sounds like a great project, one I heartily recommend to people wanting to get their feet wet. First, the basic principles of how to write an efficient prefix sum are the same as before. Definitely read [1]. Also see [2] for an exploration into how much subgroups help over threadgroup shared memory. In fact, [2] has much of the answer you seek, as it's already written in Vulkan and GLSL (compiling to SPIR-V), though it does miss a direct performance comparison to CUB. Second, doing this depends on WebGPU exposing threadgroup shared memory and subgroups operations. I'm not sure where the standard and implementations are on this; there are significant challenges. For example, subgroup shuffle is not available in DX12, though subgroup reduce operations such as WavePrefixSum are. So in general an implementation will need to do runtime detection of capabilities and choose a most-optimum kernel based on that. In theory, it should be possible to achieve comparable performance. But it's likely that WebGPU implementations haven't really been tuned for performance yet, while CUB has seen a ton of work. So, as always, the thing to do is implement and measure. [1]: https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-computing/chapter-39-parallel-prefix-sum-scan-cuda https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-co... [2]: https://cachemiss.xyz/blog/parallel-reduce-and-scan-on-the-GPU https://cachemiss.xyz/blog/parallel-reduce-and-scan-on-the-G...
m463 7y ago
When I heard the definition SIMT, it helped me understand the if / do_a / do_b concept. But your subsequent explanation helped understand what went beyond that. Thank you for the talk.
unixhero 7y ago
Really good talk, thanks a lot.

Jhsto 7y ago

Thanks for the video and slides, super interesting! One question: in one of the slides, you mention the Vulkan 1.2 / SPIR-V 1.5 Vulkan Memory model. What exactly is this and why do you think this is meaningful? Would you have any links to learn more? When I read the SPIR-V specification, it mentions that this kind of memory model exists, but it does not put any effort into explaining why it's worthwhile or how it works on a high level.

jabl 7y ago
Perhaps in a few years when we'll start to see things like CXL providing cache-coherency between main memory and GPU memory this will become important?
raphlinus 7y ago
That could be the subject of a whole other talk :) The memory model is needed to do more sophisticated communication between the parallel workgroups. The classic example is to make a queue, even a simple one such as a simple single-producer single-consumer ring buffer (for communication between a pair of workgroups). The way this works, the producer fills an element in the buffer, then bumps an index. The consumer observes the index, then reads the element from the buffer. Without a memory model, there's no way to guarantee that this ordering is preserved, and (to map this to hardware), the write of the index bump could flow through the caches, while the write of the element could still be "dirty" in the cache on the producer, so the consumer reads stale data. What the memory model does is provide the programmer an explicit way to express ordering constraints. So the write of the index bump is with "release" semantics and the read of that by the consumer is with "acquire" semantics, which guarantees that producer writes before the release are visible to consumer reads after the acquire. This unlocks a whole range of sophisticated concurrent data structures, such as concurrent hash maps, fancier memory allocation (even without a memory model, a simple allocate-only bump allocator is feasible), and so on. Thanks for the question and kind words, I hope this illuminates.
- Jhsto 7y ago
  Thanks, the answer with some examples was exactly what I was hoping to get answered! Thank you for taking the time to respond.

throwlaplace 7y ago

of the links you put on the last page (or other gpu languages you might not have included) what's the most mature way to write decent gpu code right now (even if not necessarily optimal). i'm right this moment working on a project where i'm trying to speed up a blob detection algorithm by porting it to gpu. i'm using pytorch because the primitives fit naturally but if that weren't the case i think i would be kind of at a loss for how to do it (i do not know, as of yet, how to write cuda kernels). btw the Julia link is dead

raphlinus 7y ago
It depends on what you're trying to do. If you want to get stuff done now then just get an Nvidia card and write CUDA. The tools and knowledge community are vastly ahead of other approaches. If you're doing image-like stuff, Halide is good, there's lots of stuff shipping on it. One other project I'd add is emu, which is built on wgpu. I think that's closest to the future I have in mind, but still in early stages. https://github.com/calebwin/emu https://github.com/calebwin/emu
- throwlaplace 7y ago
  okay in that case can you recommend a good tut for writing cuda code?
  - alexhutcheson 7y ago
    The official docs are pretty good. This is probably a good starting point: https://devblogs.nvidia.com/even-easier-introduction-cuda/ https://devblogs.nvidia.com/even-easier-introduction-cuda/ Nvidia also sponsored a Udacity course, but I don't have any direct experience with it: https://developer.nvidia.com/udacity-cs344-intro-parallel-programming https://developer.nvidia.com/udacity-cs344-intro-parallel-pr...
ZeroCool2u 7y ago
If you're interested in getting started writing GPU code and you're currently dealing with PyTorch a good place to start might be using NUMBA for CUDA programming. It lets you do in Python what you'd normally do in C/C++. It's great, because setup is minimal and it has some good tutorial materials. Plus, you're mostly emulating what you'd end up doing in C/C++, so conceptually the code is very similar. When I was at GTC 2018 this is one of the training sessions I went to and you can now access that here[1]. The official NUMBA for CUDA docs are great too[2]. Also, you can avoid a lot of installation/dependencies headaches by using anaconda/miniconda if you're doing this stuff locally. [1]: https://github.com/ContinuumIO/gtc2018-numba https://github.com/ContinuumIO/gtc2018-numba [2]:https://numba.pydata.org/numba-doc/latest/cuda/overview.html https://numba.pydata.org/numba-doc/latest/cuda/overview.html

sorenjan 7y ago

Do you think other alternatives to Cuda will have the same drawback as OpenCL, that Nvidia won't support it properly and therefore not be a viable option on their hardware which is dominating the market at the moment?

gmueckl 7y ago
Well, what are the chances that less vendor lock in is in nVidia's interest?
- angry_octet 7y ago
  Is it vendor lock-in if the clients want to be there? It's more like you can be in the club with the good music and hip people, or the club with the average unknown DJ and an empty dance floor.
  - bl0b 7y ago
    That makes sense for GPU-compute-in-the-cloud applications where you can pick the hardware and don't care if other hardware doesn't work, but some people want to write programs that use GPGPU that can run on anyone's machine.
raphlinus 7y ago
No, precisely because I see the emerging alternatives as being based on Vulkan and SPIR-V, which I believe have momentum far greater than OpenCL. The only question is whether a compute platform based on these standards and open source tools could be anywhere nearly as good as CUDA. I believe we're far from that today, but it is possible and there's a lot of good work in that direction.
- muro 7y ago
  Doesn't that exclude Apple?
  - 7y ago

sprash 7y ago

Halfway through you were bringing up hopes for a great new way to do 2D rendering on GPUs but then there is no show of actual implementation. Rather anticlimactic! Even decades old Cairo is beating VulKan/GPU based renderers like Skia in some tasks. Can it be that hardware intended for ML or 3D-graphics is not suited at all for 2D rendering (parallelism vs. pipelining)? How will WebGPU solve those problems? To me this talk raises more questions than it answers...

raphlinus 7y ago
It's work in progress. I was hoping to get more done on the renderer but various things have been slowing me down. But lately I've been feeling a new burst of energy, so hope to have something more to show before long. I agree that the talk raises a lot of questions. Hopefully that's a good thing, as I think they deserve the attention of curious people! Regarding the mismatch between 3D hardware and 2D rendering, we'll need hard empirical numbers, but the approach I'm taking is basically to write a software renderer (not unlike Cairo or even libart which predates it), just one that's designed for massive parallelism so it can run on the GPU. This will have performance characteristics more similar to Cairo (just faster as so much more throughput is available) than rasterization-based approaches. The latter really fall down when there's soft clipping and lots of blending, as that requires generation of intermediate texture buffers. I do address this in the talk :) I hope I will have more solid answers for some of these questions before long.
- sprash 7y ago
  I'm experimenting with GPU based 2D rendering using signed distance functions as drawing primitives for quiet a long time now. My experience is that as soon as the object tree exceeds a certain size the CPU will always beat the GPU. The only way the GPU can win is when you have high complexity drawing primitives and a low amount of objects which is exactly the opposite one usually encounters in the real world. Maybe the partitioning approach you are proposing will help but only real world implementations and benchmarks will tell.
  - 7y ago

zozbot234 7y ago

Is Vulkan really becoming viable for compute? It seems like something that's obviously worth exploring given the overlap with more traditional graphics workloads, but I have seen some claims that it's still lagging behind OpenCL wrt. some features, and that it could be good for some things but is not obviously competitive across the board. It's possible that this impression is outdated though.

raphlinus 7y ago
I think the answer to this is emerging, but I would say yes. Vulkan and SPIR-V are rapidly gaining features, while OpenCL is stagnating. Some of these features are very recent, like pointers and a memory model. Another area I'm expecting Vulkan to pull ahead is support for efficient float16 computation, which is important for ML workloads, and is increasingly well supported in hardware. One thing to keep in mind though, programming directly in Vulkan is too low-level for most users, while OpenCL is a fairly friendly API. We need tools that can compile to SPIR-V and deploy on Vulkan, but high enough level you can express your problem rather than (for example) manually placing pipeline barriers between dispatches. The lack of such tools, and having to build some of my own, is one of the reasons for the slowdown in progress on my renderer project.
- Jasper_ 7y ago
  CUDA is unfortunately very, very far ahead of Vulkan. "Graphics Compute" is quite different from CUDA at the advanced levels if you've never used it.
- pjmlp 7y ago
  In CUDA I can just use ISO C++17 (C++20 around the corner) plus CUDA extensions, or any other language that targets PTX, with very nice graphical debugging from what is going on the GPGPU. OpenCL is playing catch up with CUDA, but it requires at least version 2.0 for similar capabilities, while many vendors are still shipping 1.x drivers. Vulkan, well it seems to be the same story as OpenCL regarding compute tooling, but lets see how much love it will get from NVidia. I advise watching the GTC 2020 talks regarding CUDA and Vulkan compute.

Symmetry 7y ago

Any opinion on ROCm?

mpreda 7y ago
ROCm provides a good OpenCL 2.x implementation targeting AMD GCN. The compiler/optimizer is clang-based. It is open-source. One great thing is that ROCm OpenCL supports GCN assembly in the OpenCL source, which is handy in some corner-cases that are not properly handled by the optimizer.
VHRanger 7y ago
Doesn't support many platforms

Strilanc 7y ago

What is a reasonable lower limit on latency that I could expect from a GPU pipeline? For example, I know that if I use WebGL and issue a `readPixels` call then that seems to take on the order of 1 to 10 milliseconds no matter what I do. This suggests that if I needed to react to something in under 100 microseconds, I probably shouldn't have the computation of the reaction involve WebGL. But is that true of GPU compute in general, or just an artifact of the abstraction exposed by WebGL?

jdashg 7y ago
There's a bunch of intractable latency, but there are patterns that can help maintain pipelining, if you don't need the data immediately: https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/WebGL_best_practices#Non-blocking_async_data_downloadreadback https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/W...
- Strilanc 7y ago
  Right, but if I go closer to the metal how much of the latency is really truly intractable? 100 microseconds? 1 millisecond? 10 milliseconds?
  - bl0b 7y ago
    Can't speak for WebGL APIs, but in OpenGL (and other C++ graphics/GPGPU platforms), it is certainly possible (and necessary) to write fully interactive real-time apps that rely on a few rounds of communication between GPU and CPU code every frame. At 60 fps, you have 16 ms per frame. As long as you aren't copying huge arrays back and forth, I'd say you can budget in the 1 ms order-of-magnitude per data transfer. I don't know much of that would be avoidable if you were actually closer to the metal than the raw OpenGL C++ API call
kllrnohj 7y ago
You do have to design around the GPU typically being almost like a server. It's very fast, but also "very far" away - both in latency & in bandwidth. PCI-E 3.0 x16 is 'only' 16GB/s. That's fast for I/O, but still in the ballpark of being like I/O & not like RAM. And that's if you even get x16 electrical - x8 for GPUs is pretty common in eg. laptops where the other lanes go to things like thunderbolt 3. Integrated graphics are a different story, they often just have a direct connection to DRAM and CPU<->GPU communication is therefore super fast & low latency. In theory, anyway, if the API abstractions even let you share memory between the two. So that means even for things where GPUs should do really well at it, like summing two arrays together, doing it on the CPU can still be a lot faster if the inputs & outputs are local to the CPU.

2008guy 7y ago

What were you talking about with the socialist revolution?

mkl 7y ago
He was giving a talk to people from Jane Street [1], who make money on the stock market using OCaml (mentioned in the same bit). [1] https://en.wikipedia.org/wiki/Jane_Street_Capital https://en.wikipedia.org/wiki/Jane_Street_Capital
- 2008guy 7y ago
  Right but he basically asserted to everyone that a socialist revolution is coming to wipe them out. Is there something I don’t know? Should I be getting ready for this?
  - mkl 7y ago
    Huh? It was a joke. Explaining the joke will kill it, but here goes: He was saying that their OCaml skills would still be useful even if they couldn't make money in finance because capitalism fell over. The audience got the joke and laughed. It's nothing like what you're saying.
    - 2008guy 7y ago
      What I saw was this guy flatly asserting that a socialist revolution is coming, I thought at first he was talking about Bernie Sanders, and where there should be a punchline there is no punchline. Then everyone gives an awkward and half-cocked chuckle as one does when presented with something so bizarre.

fluffything 7y ago

I write a lot of compute kernels in CUDA, and my litmus test is prefix sum, for two reasons. First, you can implement it in pure CUDA C++, and max out the memory bandwidth of any nvidia or AMD GPU. The CUB library provides a state of the art implementation (using decoupled-lookback) that one can compare against new programming languages. Second, it is one of the most basic parallel algorithm building blocks. Many parallel algorithms use it, and many parallel algorithms are "prefix-sum-like". If I am not able to write prefix sum from scratch efficiently in your programming language / library, I can't use it. Every time someone shows a new programming language for compute, the examples provided are super basic (e.g. `map(...).fold(...)`), but I have yet to see a new programming language that can be used to implement the 2-3 most fundamental parallel algorithms from any parallel algorithms graduate course. For example, Futhark provides a prefix sum intrinsic, that just calls CUB - if you want to implement a prefix-sum-like algorithm, you are out of luck. In WGPU, it appears that prefix-sum will be an intrinsic of WHSL, which sounds like you would be out-of-luck too. You mentioned WGPU and Vulkan. Do you know how to implement prefix-sum from scratch on these? If so, do you know how the performance compare against CUB?

raphlinus 7y ago
That sounds like a great project, one I heartily recommend to people wanting to get their feet wet. First, the basic principles of how to write an efficient prefix sum are the same as before. Definitely read [1]. Also see [2] for an exploration into how much subgroups help over threadgroup shared memory. In fact, [2] has much of the answer you seek, as it's already written in Vulkan and GLSL (compiling to SPIR-V), though it does miss a direct performance comparison to CUB. Second, doing this depends on WebGPU exposing threadgroup shared memory and subgroups operations. I'm not sure where the standard and implementations are on this; there are significant challenges. For example, subgroup shuffle is not available in DX12, though subgroup reduce operations such as WavePrefixSum are. So in general an implementation will need to do runtime detection of capabilities and choose a most-optimum kernel based on that. In theory, it should be possible to achieve comparable performance. But it's likely that WebGPU implementations haven't really been tuned for performance yet, while CUB has seen a ton of work. So, as always, the thing to do is implement and measure. [1]: https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-computing/chapter-39-parallel-prefix-sum-scan-cuda https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-co... [2]: https://cachemiss.xyz/blog/parallel-reduce-and-scan-on-the-GPU https://cachemiss.xyz/blog/parallel-reduce-and-scan-on-the-G...

m463 7y ago

When I heard the definition SIMT, it helped me understand the if / do_a / do_b concept. But your subsequent explanation helped understand what went beyond that. Thank you for the talk.

unixhero 7y ago

Really good talk, thanks a lot.