4 ms·

Knowing how many registers are allocated is roughly how many are live, which is how many to write to the stack during a coroutine switch. I'm having trouble wi

Knowing how many registers are allocated is roughly how many are live, which is how many to write to the stack during a coroutine switch. I'm having trouble with the tradeoff between compiler complexity and perceived value of userspace scheduling on gpus so a means of moving work from compiler into runtime helps lower the barrier to landing the implementation. Oh, or with the work hat on, function pointers are a nuisance with kernel allocated resources, and a jump table keyed off the result of a register file size query would be a partial solution to that.

kimixa 2y ago
At any time a coroutine switch could happen, doesn't the compiler statically know how many registers would be allocated by this anyway? Register numbers aren't dynamically indexed, they're baked into the instruction stream, so there's not really any new information you can't get from the instruction pointer and a static (compiler-generated) lookup table?
- JonChesterfield 2y ago
  Yep, the compiler can do it all. However getting proof of concepts together to justify shipping the compiler complexity can go better with more work done by the runtime code.
phire 2y ago
Yes... I could see that approach being desirable on a CPU, it would make all context switches cheaper. Are we re-inventing register windows? Of course, we are on a GPU here. Skimming through the RDNA 4 ISA document [1], looks like they expect waves using the dynamic VGPRs features to keep track of how many VGPRs they have requested in a Scalar register (SGPR). Each wave always has 106 SGPRs allocated (32-bit each), so the overhead of a single SGPR isn't an issue. You wouldn't even need the jump table, as AMD allows you to index into the VGPR register file. A small loop would allow you to push all active VGPRs onto the stack. Shame they don't also allow you to index into the SGPR registers too. But GPUs aren't really designed with coroutines or context switching in mind. Modern GPUs can do it, but GPUs have deep historical assumptions that every single wave on every single core will be running the exact same code, and that 99.99% of waves will run to completion without needing a context switch. When preemptive context switches are needed (no more than once every few ms, and only with long-running compute shaders), they can probably get away with simply DMAing the whole register file to memory. Though, part of the reason this feature was implemented is that the historic assumptions are no-longer true. DXR allows each material to register a unique hit shader, so Modern GPUs need to dynamic dispatch based on the result of ray hits. [1] https://www.amd.com/content/dam/amd/en/documents/radeon-tech-docs/instruction-set-architectures/rdna4-instruction-set-architecture.pdf https://www.amd.com/content/dam/amd/en/documents/radeon-tech...
- dragontamer 2y ago
  > DXR allows each material to register a unique hit shader, so Modern GPUs need to dynamic dispatch based on the result of ray hits. That's not how it works in practice. Even with hardware accelerated raytracers (like Intel Arc). AMD systems push the hit/miss onto various buffers and pass them around. Intel systems push the entire call-stack and shuffles them around. Lets say your 256 thread-group chunk has 30% "metalic hits", 15% "diffuse hits", and the remaining 55% are misses. You cannot "build up" a new group with just one thread-group (!!!!). To efficiently run things, you'll need ~4 thread groups (aka: 1024 rays) before you can run a full 256-thread group again for hits, and you'll need ~2 thread-groups (aka: 512 rays) before you get a full 256-thread group again for misses. And finally you'll need ~7 thread-groups (aka: 1792 rays to pass through) before you have the 256-diffuse hits needed to fill up a SIMD Unit. In all cases, you need to dynamically grow a buffer and "build up" enough parallelism before running the recursive hits (or miss) handlers. The devil is in the details. Intel has very dedicated and specialized accelerators that moves the stacks around (!!!!) so that all groups remain fully utilized. I believe AMD's implementation is "just" an append buffer followed by a consume buffer, simple enough really. Probably inside of shared memory but who knows what the full implementation details are. (The AMD systems have documented ISAs so we know what instructions are available. AMD's "raytracers" are BVH tree traversal accelerators but don't seem to have stack-manipulation or stack-movements like Intel's raytracing implementation)
dragontamer 2y ago
Coroutines aren't really a thing in modern GPUs. You might get similar effects with work graphs, CUDA dynamic parallelism or whatnot. But most coroutine patterns have significant branch divergence that probably is grossly inefficient in GPU land. The GPU equivalent to coroutines would be to save off a continuation structure to a append buffer, and then startup a new GPU call with that buffer now turned into a consume buffer (which is how Raytracing circa 2018ish worked in Blender, before deficated Raytracing hardware was a thing) You need to consolidate all of the divergent threads and coalesce them into converged threads to benefit from SIMD architecture. ------- EDIT: Remember that in a 1920 x 1080p pixel screen has 2073600 pixels, which becomes 2073600 threads for 1-ray-per-pixel typical GPU implementation. With so many threads, it doesn't make sense to "immediately" call coroutines. Instead, it makes far more sense to consolidate and coalesce threads together (ex: group up "hit" raytraces vs "miss" raytraces, which are handled differently).
- JonChesterfield 2y ago
  It makes a lot more sense when programming it as a vector machine. The divergence/convergence stuff is the cuda/simt model, there's no need to use that on amdgpu. Branches are cheap(ish) when they're done on the scalar unit. Coroutines aren't currently a thing on amdgpu but I think they should be.
  - dragontamer 2y ago
    > The divergence/convergence stuff is the cuda/simt model Even on NVidia, you're "allowed" to diverge and converge. But its not efficient. Optimal NVidia coding will force more convergence than divergence. That's innate to GPU architecture. Its more efficient to run 32-at-a-time per NVidia warp, than a diverged 8-at-a-time warp. Yes, NVidia _CAN_ diverge and properly execute a subwarp of 8-at-a-time per clocktick... including with complex atomics and all that. But running a full 32-at-a-time warp is 400% the speed because its ALWAYS better to do more per clock tick than less-per-clock tick.

kimixa 2y ago

At any time a coroutine switch could happen, doesn't the compiler statically know how many registers would be allocated by this anyway? Register numbers aren't dynamically indexed, they're baked into the instruction stream, so there's not really any new information you can't get from the instruction pointer and a static (compiler-generated) lookup table?

JonChesterfield 2y ago
Yep, the compiler can do it all. However getting proof of concepts together to justify shipping the compiler complexity can go better with more work done by the runtime code.

phire 2y ago

Yes... I could see that approach being desirable on a CPU, it would make all context switches cheaper. Are we re-inventing register windows? Of course, we are on a GPU here. Skimming through the RDNA 4 ISA document [1], looks like they expect waves using the dynamic VGPRs features to keep track of how many VGPRs they have requested in a Scalar register (SGPR). Each wave always has 106 SGPRs allocated (32-bit each), so the overhead of a single SGPR isn't an issue. You wouldn't even need the jump table, as AMD allows you to index into the VGPR register file. A small loop would allow you to push all active VGPRs onto the stack. Shame they don't also allow you to index into the SGPR registers too. But GPUs aren't really designed with coroutines or context switching in mind. Modern GPUs can do it, but GPUs have deep historical assumptions that every single wave on every single core will be running the exact same code, and that 99.99% of waves will run to completion without needing a context switch. When preemptive context switches are needed (no more than once every few ms, and only with long-running compute shaders), they can probably get away with simply DMAing the whole register file to memory. Though, part of the reason this feature was implemented is that the historic assumptions are no-longer true. DXR allows each material to register a unique hit shader, so Modern GPUs need to dynamic dispatch based on the result of ray hits. [1] https://www.amd.com/content/dam/amd/en/documents/radeon-tech-docs/instruction-set-architectures/rdna4-instruction-set-architecture.pdf https://www.amd.com/content/dam/amd/en/documents/radeon-tech...

dragontamer 2y ago
> DXR allows each material to register a unique hit shader, so Modern GPUs need to dynamic dispatch based on the result of ray hits. That's not how it works in practice. Even with hardware accelerated raytracers (like Intel Arc). AMD systems push the hit/miss onto various buffers and pass them around. Intel systems push the entire call-stack and shuffles them around. Lets say your 256 thread-group chunk has 30% "metalic hits", 15% "diffuse hits", and the remaining 55% are misses. You cannot "build up" a new group with just one thread-group (!!!!). To efficiently run things, you'll need ~4 thread groups (aka: 1024 rays) before you can run a full 256-thread group again for hits, and you'll need ~2 thread-groups (aka: 512 rays) before you get a full 256-thread group again for misses. And finally you'll need ~7 thread-groups (aka: 1792 rays to pass through) before you have the 256-diffuse hits needed to fill up a SIMD Unit. In all cases, you need to dynamically grow a buffer and "build up" enough parallelism before running the recursive hits (or miss) handlers. The devil is in the details. Intel has very dedicated and specialized accelerators that moves the stacks around (!!!!) so that all groups remain fully utilized. I believe AMD's implementation is "just" an append buffer followed by a consume buffer, simple enough really. Probably inside of shared memory but who knows what the full implementation details are. (The AMD systems have documented ISAs so we know what instructions are available. AMD's "raytracers" are BVH tree traversal accelerators but don't seem to have stack-manipulation or stack-movements like Intel's raytracing implementation)

dragontamer 2y ago

Coroutines aren't really a thing in modern GPUs. You might get similar effects with work graphs, CUDA dynamic parallelism or whatnot. But most coroutine patterns have significant branch divergence that probably is grossly inefficient in GPU land. The GPU equivalent to coroutines would be to save off a continuation structure to a append buffer, and then startup a new GPU call with that buffer now turned into a consume buffer (which is how Raytracing circa 2018ish worked in Blender, before deficated Raytracing hardware was a thing) You need to consolidate all of the divergent threads and coalesce them into converged threads to benefit from SIMD architecture. ------- EDIT: Remember that in a 1920 x 1080p pixel screen has 2073600 pixels, which becomes 2073600 threads for 1-ray-per-pixel typical GPU implementation. With so many threads, it doesn't make sense to "immediately" call coroutines. Instead, it makes far more sense to consolidate and coalesce threads together (ex: group up "hit" raytraces vs "miss" raytraces, which are handled differently).

JonChesterfield 2y ago
It makes a lot more sense when programming it as a vector machine. The divergence/convergence stuff is the cuda/simt model, there's no need to use that on amdgpu. Branches are cheap(ish) when they're done on the scalar unit. Coroutines aren't currently a thing on amdgpu but I think they should be.
- dragontamer 2y ago
  > The divergence/convergence stuff is the cuda/simt model Even on NVidia, you're "allowed" to diverge and converge. But its not efficient. Optimal NVidia coding will force more convergence than divergence. That's innate to GPU architecture. Its more efficient to run 32-at-a-time per NVidia warp, than a diverged 8-at-a-time warp. Yes, NVidia _CAN_ diverge and properly execute a subwarp of 8-at-a-time per clocktick... including with complex atomics and all that. But running a full 32-at-a-time warp is 400% the speed because its ALWAYS better to do more per clock tick than less-per-clock tick.