4 ms·

> with SIMD-instructions You assume so, for Volta onwards that's not true. Each SIMT thread on NVIDIA GPUs for Volta onwards has its own instruction pointer.

by my123 6y ago

> with SIMD-instructions You assume so, for Volta onwards that's not true. Each SIMT thread on NVIDIA GPUs for Volta onwards has its own instruction pointer. And yes, it's scalar instructions at the ISA level.

dragontamer 6y ago
> You assume so, for Volta onwards that's not true. https://arxiv.org/pdf/1804.06826.pdf https://arxiv.org/pdf/1804.06826.pdf NVidia's SASS on Volta is pretty clearly SIMD-instructions. The PTX "virtual machine assembly" is fully documented. SASS is not documented, but it ain't a secret either. SASS follows closely with PTX, with exception of some "barrier bits" that seem to track dependencies (probably compiler-managed read/write dependency chains). > Each SIMT thread on NVIDIA GPUs for Volta onwards has its own instruction pointer. And yes, it's scalar instructions at the ISA level. When your "scalar" instruction executes 32-threads in parallel (subject to an execution mask), that's called SIMD. The instruction pointer is to resolve deadlock conditions with locks, and the 32-wide SIMD cores will execute one-at-a-time to prevent deadlocks. But your goal as a GPU programmer is to get as much 32-wide execution going on as possible. 1-at-a-time serialization is very, very bad for performance. Its POSSIBLE to do on Volta, but highly recommended you stay away from that cornercase (you lose over 95% of your performance).
- my123 6y ago
  Apart from tensor cores and some compressed instructions on small data types, they are not SIMD instructions. Each individual SIMT lane can actually diverge from the other on the underlying hardware. Active threads are dynamically mapped to SIMT units. You can use __syncwarp() to force reconvergence, to stall until all the threads in a warp are at the same location. The disassembled code targeting the underlying ISA for a given CUDA kernel is also easily accessible through nvdisasm.
  - dragontamer 6y ago
    A 32-wide SIMD core executing 16 or 8 or 1-thread at a time is just SIMD execution that's running at 50%, 25%, or 3% utilization. See figure 22 and figure 23 in: https://images.nvidia.com/content/volta-architecture/pdf/volta-architecture-whitepaper.pdf https://images.nvidia.com/content/volta-architecture/pdf/vol... Thread divergence is bad: very very bad. Running 2x, 4x, or 32x slower (or less parallel technically). You want to avoid those situations as much as possible. What NVidia noticed is that thread divergence is necessary for many classical locking algorithms: where serialized code must execute one-at-a-time to run an algorithm correctly. Under these conditions, tracking the instruction pointer, and turning off 97% of your cores to execute 1-at-a-time (instead of 32-at-a-time SIMD) is done. That's WHY __syncwarp() exists, so that you can return to 32-at-a-time execution as soon as possible. Its not always possible for the compiler to figure it out, so the programmer can put a __syncwarp() as a compiler hint that 32-at-a-time is safe again.
    - my123 6y ago
      You claimed that SASS was a SIMD instruction set, which is false. (through "NVidia's SASS on Volta is pretty clearly SIMD-instructions") Yes, thread divergence on GPUs can come with performance downsides and is a tool to use carefully.
      - dragontamer 6y ago
        > You claimed that SASS was a SIMD instruction set, which is false. Well, they execute 32-at-a-time SIMD, do they not? There are tricks to split up the execution mask and execute 1-at-a-time, 2-at-a-time, 4, 8, 16, or 31 at a time based on if-statements or thread-locks or whatever. But a "SASS" instruction of "R1 = R2 + R3" is implicitly executed across a 32-wide warp if its execution mask is set as such. ---------- EDIT: Lets put it this way: If you see a "SASS" assembly for R1 = R2 + R3 in isolation, how many adds take place on that clock tick? Somewhere between 1 add, and 32-adds. No more than 32-at-a-time. Seems pretty SIMD to me.
        my123 6y ago
        There is no execution mask from the ISA perspective, each thread has its own instruction pointer, and the hardware deals with which threads get executed, instead of a mask being set/unset. I don't know why you continue to argue this...

by my123 6y ago

dragontamer 6y ago

> You assume so, for Volta onwards that's not true. https://arxiv.org/pdf/1804.06826.pdf https://arxiv.org/pdf/1804.06826.pdf NVidia's SASS on Volta is pretty clearly SIMD-instructions. The PTX "virtual machine assembly" is fully documented. SASS is not documented, but it ain't a secret either. SASS follows closely with PTX, with exception of some "barrier bits" that seem to track dependencies (probably compiler-managed read/write dependency chains). > Each SIMT thread on NVIDIA GPUs for Volta onwards has its own instruction pointer. And yes, it's scalar instructions at the ISA level. When your "scalar" instruction executes 32-threads in parallel (subject to an execution mask), that's called SIMD. The instruction pointer is to resolve deadlock conditions with locks, and the 32-wide SIMD cores will execute one-at-a-time to prevent deadlocks. But your goal as a GPU programmer is to get as much 32-wide execution going on as possible. 1-at-a-time serialization is very, very bad for performance. Its POSSIBLE to do on Volta, but highly recommended you stay away from that cornercase (you lose over 95% of your performance).

my123 6y ago
Apart from tensor cores and some compressed instructions on small data types, they are not SIMD instructions. Each individual SIMT lane can actually diverge from the other on the underlying hardware. Active threads are dynamically mapped to SIMT units. You can use __syncwarp() to force reconvergence, to stall until all the threads in a warp are at the same location. The disassembled code targeting the underlying ISA for a given CUDA kernel is also easily accessible through nvdisasm.
- dragontamer 6y ago
  A 32-wide SIMD core executing 16 or 8 or 1-thread at a time is just SIMD execution that's running at 50%, 25%, or 3% utilization. See figure 22 and figure 23 in: https://images.nvidia.com/content/volta-architecture/pdf/volta-architecture-whitepaper.pdf https://images.nvidia.com/content/volta-architecture/pdf/vol... Thread divergence is bad: very very bad. Running 2x, 4x, or 32x slower (or less parallel technically). You want to avoid those situations as much as possible. What NVidia noticed is that thread divergence is necessary for many classical locking algorithms: where serialized code must execute one-at-a-time to run an algorithm correctly. Under these conditions, tracking the instruction pointer, and turning off 97% of your cores to execute 1-at-a-time (instead of 32-at-a-time SIMD) is done. That's WHY __syncwarp() exists, so that you can return to 32-at-a-time execution as soon as possible. Its not always possible for the compiler to figure it out, so the programmer can put a __syncwarp() as a compiler hint that 32-at-a-time is safe again.
  - my123 6y ago
    You claimed that SASS was a SIMD instruction set, which is false. (through "NVidia's SASS on Volta is pretty clearly SIMD-instructions") Yes, thread divergence on GPUs can come with performance downsides and is a tool to use carefully.
    - dragontamer 6y ago
      > You claimed that SASS was a SIMD instruction set, which is false. Well, they execute 32-at-a-time SIMD, do they not? There are tricks to split up the execution mask and execute 1-at-a-time, 2-at-a-time, 4, 8, 16, or 31 at a time based on if-statements or thread-locks or whatever. But a "SASS" instruction of "R1 = R2 + R3" is implicitly executed across a 32-wide warp if its execution mask is set as such. ---------- EDIT: Lets put it this way: If you see a "SASS" assembly for R1 = R2 + R3 in isolation, how many adds take place on that clock tick? Somewhere between 1 add, and 32-adds. No more than 32-at-a-time. Seems pretty SIMD to me.
      - my123 6y ago
        There is no execution mask from the ISA perspective, each thread has its own instruction pointer, and the hardware deals with which threads get executed, instead of a mask being set/unset. I don't know why you continue to argue this...