6 ms·

Dynamic Register Allocation on AMD's RDNA 4 GPU Architecture

127 points by ingve 2y ago

JonChesterfield 2y ago
That's a really good trick. New to me. Anyone know a reasonable way to do the query equivalent? Not allocate more, ask how many are currently available.
- djmips 2y ago
  On AMD ,not available to users yet AFAIK. Only available to the driver and currently targeted at ray tracing? On Nvidia you have to be writing in the PTX language to have access to something similar.
- dragontamer 2y ago
  Why bother? If the next assembly line says to use register#200 (but you failed because you only have 160 registers), what's the processor supposed to do? The only hope is to call for more registers and lock until those registers are available. There's no way to progress otherwise.
  - im3w1l 2y ago
    Dynamically switch algorithms between one that is faster but has more register pressure and a slower one with less pressure. Lowering numerical precision might be on the table too.
    - wtallis 2y ago
      And what, refill the instruction cache in less time than it takes to wait for more registers to become available?
  - JonChesterfield 2y ago
    Knowing how many registers are allocated is roughly how many are live, which is how many to write to the stack during a coroutine switch. I'm having trouble with the tradeoff between compiler complexity and perceived value of userspace scheduling on gpus so a means of moving work from compiler into runtime helps lower the barrier to landing the implementation. Oh, or with the work hat on, function pointers are a nuisance with kernel allocated resources, and a jump table keyed off the result of a register file size query would be a partial solution to that.
    - kimixa 2y ago
      At any time a coroutine switch could happen, doesn't the compiler statically know how many registers would be allocated by this anyway? Register numbers aren't dynamically indexed, they're baked into the instruction stream, so there's not really any new information you can't get from the instruction pointer and a static (compiler-generated) lookup table?
      - JonChesterfield 2y ago
        Yep, the compiler can do it all. However getting proof of concepts together to justify shipping the compiler complexity can go better with more work done by the runtime code.
    - phire 2y ago
      Yes... I could see that approach being desirable on a CPU, it would make all context switches cheaper. Are we re-inventing register windows? Of course, we are on a GPU here. Skimming through the RDNA 4 ISA document [1], looks like they expect waves using the dynamic VGPRs features to keep track of how many VGPRs they have requested in a Scalar register (SGPR). Each wave always has 106 SGPRs allocated (32-bit each), so the overhead of a single SGPR isn't an issue. You wouldn't even need the jump table, as AMD allows you to index into the VGPR register file. A small loop would allow you to push all active VGPRs onto the stack. Shame they don't also allow you to index into the SGPR registers too. But GPUs aren't really designed with coroutines or context switching in mind. Modern GPUs can do it, but GPUs have deep historical assumptions that every single wave on every single core will be running the exact same code, and that 99.99% of waves will run to completion without needing a context switch. When preemptive context switches are needed (no more than once every few ms, and only with long-running compute shaders), they can probably get away with simply DMAing the whole register file to memory. Though, part of the reason this feature was implemented is that the historic assumptions are no-longer true. DXR allows each material to register a unique hit shader, so Modern GPUs need to dynamic dispatch based on the result of ray hits. [1] https://www.amd.com/content/dam/amd/en/documents/radeon-tech-docs/instruction-set-architectures/rdna4-instruction-set-architecture.pdf https://www.amd.com/content/dam/amd/en/documents/radeon-tech...
- Someone 2y ago
  I don’t know whether it’s reasonable, but > Allocation requests don’t always succeed. s_alloc_vgpr sets the Scalar Condition Code (SCC) to indicate success, or clears it on failure. So, you can do a binary search: - set result to zero. - try to allocate 512. If that succeeds, add 512 to result - try to allocate 256. If that succeeds, add 256 to result - try to allocate 128. If that succeeds, add 128 to result ... - try to allocate 1. If that succeeds, add 1 to result (you’ll want to stop at the granularity at which registers are allocated) - free ’result’ registers (This isn’t race-free)
  - Tuna-Fish 2y ago
    Note that: 1. Registers are not allocated individually, but in clusters of 32, and... 2. The max registers a single thread can access is 8 clusters (256 registers). The minimum is 1 cluster. You don't really need much of a binary search for this.
  - dragontamer 2y ago
    The issue is there's plan b: spill your data to L1 cache. Whatever plan you have needs to be far faster than L1 cache to even make sense. I can imagine some very heavy register-pressure situations where registers are much better than cache accesses.
- dagmx 2y ago
  Given the speed of occupancy changes, you’d almost always fall afoul of TOCTOU on this.
monocasa 2y ago
Huh, I would have expected the request for additional registers to be linked into the resource request scoreboarding you see for texture accesses, rt hardware requests, etc. Busy waiting on condition code seems non optimal, and opening the window for cases where one or all shader programs on core can no longer make progress.
- phire 2y ago
  The feature was clearly designed for the ray-tracing use case, where the waves are independent and really shouldn't run into deadlocks as long as one wave can continue. That's why they have the deadlock avoidance mode, it guarantees one wave will always continues. I believe they are using this to store the BVH traversal stack in registers, with allocation size depending on traversal depth. Eventually all the rays in the wave will hit or miss and free those registers. Busy waiting is only the naive solution. I bet they went with the software based solution so they can experiment with other approaches. Such as dynamically switching to the previous approach of a stack in shared-memory when registers run out. Or maybe they can sort the rays, moving the long-running rays out, grouping them together into waves dedicated for deep traversal.
  - phire 2y ago
    Actually, I'm wrong. AMD's deep dive [1] says register use is low during BVH traversal, so they are still using shared memory for the stack. According to the ISA doc [2] they have a special mode for the stack that overwrites older entries instead of overflowing and a fallback stackless mode. So the register allocation is actually increased when invoking the hit shader, allowing each hit shader to use a different number of VGPRs, spinning is probably the most logical approach. They probably don't even need the deadlock avoidance mode with their current implementation, which seems to allocate the full set of registers when calling the hit shader and only free them at the end. [1] https://www.techpowerup.com/review/amd-radeon-rx-9070-series-technical-deep-dive/3.html https://www.techpowerup.com/review/amd-radeon-rx-9070-series... [2] See 12.5.3. DS Stack Operations for Ray Tracing - https://www.amd.com/content/dam/amd/en/documents/radeon-tech-docs/instruction-set-architectures/rdna4-instruction-set-architecture.pdf https://www.amd.com/content/dam/amd/en/documents/radeon-tech...
kmeisthax 2y ago
Dumb question, but if register contention deadlock is possible, what would stop the shaders or hardware from, say, detecting the deadlock and spilling one or more threads to local memory?
- achierius 2y ago
  To answer your questions by parts: Shaders: the shader compiler does go to great lengths to avoid register bank conflicts of all kinds. Hardware: you generally don't want to have that much complexity in a GPU pipeline, especially not in the front end -- these are very simply machines, no OpO execution, very little prediction, and so their power comes from width, not depth. If you make each core a little more complex, you lose out because you now need to spend area that would have otherwise been spent on more compute units.
- dragontamer 2y ago
  If spilling to L1 cache was acceptable, then don't use this feature. The compiler automatically spills your data to memory / the stack. This exists because the GPU assembly programmer apparently wishes to provide a performance guarantee of some kind.
- Symmetry 2y ago
  And, then in theory, restoring the thread when the requested registers become available. But is the spilling/restoring registers hardware something that would see much other use? In some code hardware accelerated stack management can be a real advantage. In other code its just not flexible enough to make sense. And in some function calls are rare enough to make it not worth the investment. I sort of expect that more code running on a GPU is in the third category but I could image that I'm wrong and that it would be in the first. But if not the investment in those mechanisms might not be worth it.

127 points by ingve 2y ago

That's a really good trick. New to me. Anyone know a reasonable way to do the query equivalent? Not allocate more, ask how many are currently available.

djmips 2y ago
On AMD ,not available to users yet AFAIK. Only available to the driver and currently targeted at ray tracing? On Nvidia you have to be writing in the PTX language to have access to something similar.
dragontamer 2y ago
Why bother? If the next assembly line says to use register#200 (but you failed because you only have 160 registers), what's the processor supposed to do? The only hope is to call for more registers and lock until those registers are available. There's no way to progress otherwise.
- im3w1l 2y ago
  Dynamically switch algorithms between one that is faster but has more register pressure and a slower one with less pressure. Lowering numerical precision might be on the table too.
  - wtallis 2y ago
    And what, refill the instruction cache in less time than it takes to wait for more registers to become available?
- JonChesterfield 2y ago
  Knowing how many registers are allocated is roughly how many are live, which is how many to write to the stack during a coroutine switch. I'm having trouble with the tradeoff between compiler complexity and perceived value of userspace scheduling on gpus so a means of moving work from compiler into runtime helps lower the barrier to landing the implementation. Oh, or with the work hat on, function pointers are a nuisance with kernel allocated resources, and a jump table keyed off the result of a register file size query would be a partial solution to that.
  - kimixa 2y ago
    At any time a coroutine switch could happen, doesn't the compiler statically know how many registers would be allocated by this anyway? Register numbers aren't dynamically indexed, they're baked into the instruction stream, so there's not really any new information you can't get from the instruction pointer and a static (compiler-generated) lookup table?
    - JonChesterfield 2y ago
      Yep, the compiler can do it all. However getting proof of concepts together to justify shipping the compiler complexity can go better with more work done by the runtime code.
  - phire 2y ago
    Yes... I could see that approach being desirable on a CPU, it would make all context switches cheaper. Are we re-inventing register windows? Of course, we are on a GPU here. Skimming through the RDNA 4 ISA document [1], looks like they expect waves using the dynamic VGPRs features to keep track of how many VGPRs they have requested in a Scalar register (SGPR). Each wave always has 106 SGPRs allocated (32-bit each), so the overhead of a single SGPR isn't an issue. You wouldn't even need the jump table, as AMD allows you to index into the VGPR register file. A small loop would allow you to push all active VGPRs onto the stack. Shame they don't also allow you to index into the SGPR registers too. But GPUs aren't really designed with coroutines or context switching in mind. Modern GPUs can do it, but GPUs have deep historical assumptions that every single wave on every single core will be running the exact same code, and that 99.99% of waves will run to completion without needing a context switch. When preemptive context switches are needed (no more than once every few ms, and only with long-running compute shaders), they can probably get away with simply DMAing the whole register file to memory. Though, part of the reason this feature was implemented is that the historic assumptions are no-longer true. DXR allows each material to register a unique hit shader, so Modern GPUs need to dynamic dispatch based on the result of ray hits. [1] https://www.amd.com/content/dam/amd/en/documents/radeon-tech-docs/instruction-set-architectures/rdna4-instruction-set-architecture.pdf https://www.amd.com/content/dam/amd/en/documents/radeon-tech...
Someone 2y ago
I don’t know whether it’s reasonable, but > Allocation requests don’t always succeed. s_alloc_vgpr sets the Scalar Condition Code (SCC) to indicate success, or clears it on failure. So, you can do a binary search: - set result to zero. - try to allocate 512. If that succeeds, add 512 to result - try to allocate 256. If that succeeds, add 256 to result - try to allocate 128. If that succeeds, add 128 to result ... - try to allocate 1. If that succeeds, add 1 to result (you’ll want to stop at the granularity at which registers are allocated) - free ’result’ registers (This isn’t race-free)
- Tuna-Fish 2y ago
  Note that: 1. Registers are not allocated individually, but in clusters of 32, and... 2. The max registers a single thread can access is 8 clusters (256 registers). The minimum is 1 cluster. You don't really need much of a binary search for this.
- dragontamer 2y ago
  The issue is there's plan b: spill your data to L1 cache. Whatever plan you have needs to be far faster than L1 cache to even make sense. I can imagine some very heavy register-pressure situations where registers are much better than cache accesses.
dagmx 2y ago
Given the speed of occupancy changes, you’d almost always fall afoul of TOCTOU on this.

monocasa 2y ago

Huh, I would have expected the request for additional registers to be linked into the resource request scoreboarding you see for texture accesses, rt hardware requests, etc. Busy waiting on condition code seems non optimal, and opening the window for cases where one or all shader programs on core can no longer make progress.

phire 2y ago
The feature was clearly designed for the ray-tracing use case, where the waves are independent and really shouldn't run into deadlocks as long as one wave can continue. That's why they have the deadlock avoidance mode, it guarantees one wave will always continues. I believe they are using this to store the BVH traversal stack in registers, with allocation size depending on traversal depth. Eventually all the rays in the wave will hit or miss and free those registers. Busy waiting is only the naive solution. I bet they went with the software based solution so they can experiment with other approaches. Such as dynamically switching to the previous approach of a stack in shared-memory when registers run out. Or maybe they can sort the rays, moving the long-running rays out, grouping them together into waves dedicated for deep traversal.
- phire 2y ago
  Actually, I'm wrong. AMD's deep dive [1] says register use is low during BVH traversal, so they are still using shared memory for the stack. According to the ISA doc [2] they have a special mode for the stack that overwrites older entries instead of overflowing and a fallback stackless mode. So the register allocation is actually increased when invoking the hit shader, allowing each hit shader to use a different number of VGPRs, spinning is probably the most logical approach. They probably don't even need the deadlock avoidance mode with their current implementation, which seems to allocate the full set of registers when calling the hit shader and only free them at the end. [1] https://www.techpowerup.com/review/amd-radeon-rx-9070-series-technical-deep-dive/3.html https://www.techpowerup.com/review/amd-radeon-rx-9070-series... [2] See 12.5.3. DS Stack Operations for Ray Tracing - https://www.amd.com/content/dam/amd/en/documents/radeon-tech-docs/instruction-set-architectures/rdna4-instruction-set-architecture.pdf https://www.amd.com/content/dam/amd/en/documents/radeon-tech...

kmeisthax 2y ago

Dumb question, but if register contention deadlock is possible, what would stop the shaders or hardware from, say, detecting the deadlock and spilling one or more threads to local memory?

achierius 2y ago
To answer your questions by parts: Shaders: the shader compiler does go to great lengths to avoid register bank conflicts of all kinds. Hardware: you generally don't want to have that much complexity in a GPU pipeline, especially not in the front end -- these are very simply machines, no OpO execution, very little prediction, and so their power comes from width, not depth. If you make each core a little more complex, you lose out because you now need to spend area that would have otherwise been spent on more compute units.
dragontamer 2y ago
If spilling to L1 cache was acceptable, then don't use this feature. The compiler automatically spills your data to memory / the stack. This exists because the GPU assembly programmer apparently wishes to provide a performance guarantee of some kind.
Symmetry 2y ago
And, then in theory, restoring the thread when the requested registers become available. But is the spilling/restoring registers hardware something that would see much other use? In some code hardware accelerated stack management can be a real advantage. In other code its just not flexible enough to make sense. And in some function calls are rare enough to make it not worth the investment. I sort of expect that more code running on a GPU is in the third category but I could image that I'm wrong and that it would be in the first. But if not the investment in those mechanisms might not be worth it.