3 ms·

Honestly, this is beyond my usage and understanding. But I really appreciate such sharing findings and improvements so that everyone can benefit from them. It's

Honestly, this is beyond my usage and understanding. But I really appreciate such sharing findings and improvements so that everyone can benefit from them. It's a refreshment.

greenavocado 2y ago
FFMA (Fused Floating-point Multiply-Add) is a fundamental GPU instruction that performs D = A*B + C in a single operation. This instruction is critical for matrix multiplication and deep learning workloads. In NVIDIA's SASS (Streaming Assembly), FFMA instructions are encoded as 64-bit or 128-bit instructions with various control bits that determine their exact behavior. When the yield bit is set the bit tells the warp scheduler that the current warp can yield execution after this instruction. The hardware can then schedule a different warp to execute, potentially hiding latency. GPUs achieve high throughput through massive parallelism. When one warp stalls (e.g., waiting for memory), others can proceed. The yield bit creates explicit opportunities for the scheduler to switch warps. This bit indicates whether the source registers can be reused immediately in subsequent operations. When the yield bit is set, the reuse bit must be cleared. If a warp yields, it might not be the next one to execute. Another warp might modify the register file state. The hardware cannot guarantee register values will remain unchanged across yields. By setting the yield bit in an alternating pattern across FFMA instructions, the compiler creates explicit scheduling points where other warps can make progress. When modifying the yield bit, they also had to clear the reuse bit for affected instructions to maintain correctness. This modification specifically helps overlap two types of operations: MMA (Matrix Multiply-Accumulate) instructions: Heavy compute operations that form the core of matrix multiplication, and Promotion FFMA instructions: Operations that convert between precision formats (likely FP8 to higher precision for accumulation) FP8 (8-bit floating point) GEMM operations have specific characteristics that make this optimization particularly effective. FP8 calculations typically require conversion to higher precision for accumulation and back, creating additional FFMA operations. FP8 reduces memory bandwidth requirements but creates complex computation patterns with promotion/demotion operations. The mention of "fine-grained scaling" suggests these are operations where precision is carefully managed at multiple points in the calculation. The yield bit manipulation creates a more optimal interleaving of compute operations and format conversions, allowing the GPU to utilize its execution units more efficiently. Without this optimization, the warp scheduler might not find natural opportunities to switch between warps, leading to underutilization of compute resources.
- fotta 2y ago
  They should call the warp that is yielded to the weft.
  - dekhn 2y ago
    No, that doesn't make sense; both the yielder and yieldee are warps, the PC is the weft (approximately).
  - CamperBob2 2y ago
    Or the woof, an amusing older term.
- loginx 2y ago
  I'm gonna ask my chatgpt to write like you ;) I went from understanding none of it, to everything making sense. Thanks!
- jarbus 2y ago
  This is crazy insightful, thanks! I’d really love to learn how to get to this level of understanding, but can’t seem to figure out what curriculum I’d follow where I’d end up with this level of technical competence.
  - apples_oranges 2y ago
    Perhaps I know as much as you, but to begin, I would dive into CUDA and running code on GPUs.
  - randomNumber7 2y ago
    You need to understand how the gpu architecture works on a abstract level. Try to understand the SIMT (Single Instruction Multiple Threads) principle. Doing some shader programming or writing a cuda kernel could be a nice exercise. In a nutshell, if you want to add two vectors with hundred elements, instead of looping from 0 to 99 you would call a function called "kernel" (or "shader" in graphics programming) 100 times and pass it different indices. Then research how it is realized on the hardware with "warp"s or "wavefront"s (on AMD i think). How the cache works is also very important here. Sadly the information on the internet is relatively sparse here.

Honestly, this is beyond my usage and understanding. But I really appreciate such sharing findings and improvements so that everyone can benefit from them. It's a refreshment.

greenavocado 2y ago

FFMA (Fused Floating-point Multiply-Add) is a fundamental GPU instruction that performs D = A*B + C in a single operation. This instruction is critical for matrix multiplication and deep learning workloads. In NVIDIA's SASS (Streaming Assembly), FFMA instructions are encoded as 64-bit or 128-bit instructions with various control bits that determine their exact behavior. When the yield bit is set the bit tells the warp scheduler that the current warp can yield execution after this instruction. The hardware can then schedule a different warp to execute, potentially hiding latency. GPUs achieve high throughput through massive parallelism. When one warp stalls (e.g., waiting for memory), others can proceed. The yield bit creates explicit opportunities for the scheduler to switch warps. This bit indicates whether the source registers can be reused immediately in subsequent operations. When the yield bit is set, the reuse bit must be cleared. If a warp yields, it might not be the next one to execute. Another warp might modify the register file state. The hardware cannot guarantee register values will remain unchanged across yields. By setting the yield bit in an alternating pattern across FFMA instructions, the compiler creates explicit scheduling points where other warps can make progress. When modifying the yield bit, they also had to clear the reuse bit for affected instructions to maintain correctness. This modification specifically helps overlap two types of operations: MMA (Matrix Multiply-Accumulate) instructions: Heavy compute operations that form the core of matrix multiplication, and Promotion FFMA instructions: Operations that convert between precision formats (likely FP8 to higher precision for accumulation) FP8 (8-bit floating point) GEMM operations have specific characteristics that make this optimization particularly effective. FP8 calculations typically require conversion to higher precision for accumulation and back, creating additional FFMA operations. FP8 reduces memory bandwidth requirements but creates complex computation patterns with promotion/demotion operations. The mention of "fine-grained scaling" suggests these are operations where precision is carefully managed at multiple points in the calculation. The yield bit manipulation creates a more optimal interleaving of compute operations and format conversions, allowing the GPU to utilize its execution units more efficiently. Without this optimization, the warp scheduler might not find natural opportunities to switch between warps, leading to underutilization of compute resources.

fotta 2y ago
They should call the warp that is yielded to the weft.
- dekhn 2y ago
  No, that doesn't make sense; both the yielder and yieldee are warps, the PC is the weft (approximately).
- CamperBob2 2y ago
  Or the woof, an amusing older term.
loginx 2y ago
I'm gonna ask my chatgpt to write like you ;) I went from understanding none of it, to everything making sense. Thanks!
jarbus 2y ago
This is crazy insightful, thanks! I’d really love to learn how to get to this level of understanding, but can’t seem to figure out what curriculum I’d follow where I’d end up with this level of technical competence.
- apples_oranges 2y ago
  Perhaps I know as much as you, but to begin, I would dive into CUDA and running code on GPUs.
- randomNumber7 2y ago
  You need to understand how the gpu architecture works on a abstract level. Try to understand the SIMT (Single Instruction Multiple Threads) principle. Doing some shader programming or writing a cuda kernel could be a nice exercise. In a nutshell, if you want to add two vectors with hundred elements, instead of looping from 0 to 99 you would call a function called "kernel" (or "shader" in graphics programming) 100 times and pass it different indices. Then research how it is realized on the hardware with "warp"s or "wavefront"s (on AMD i think). How the cache works is also very important here. Sadly the information on the internet is relatively sparse here.