3 ms·

So first off, GPUs are MIMD, in that each compute-unit / symmetric multiprocessor is independent of one another. NVidia's 3xxx line of GPUs are further 2-instr

So first off, GPUs are MIMD, in that each compute-unit / symmetric multiprocessor is independent of one another. NVidia's 3xxx line of GPUs are further 2-instructions per clock tick / superscalar, which is more "traditional" MIMD, in that the cores are now doing 2 different things per clock tick on top of all the SIMD-goodness they're doing. AMD is mostly SIMD, but arguably MIMD from the perspective that it could be executing a "scalar" instruction (branch / loop instructions) in parallel with a "vector" instruction (add/subtract/multiply) ---------- So we can say that the assembly-language model of NVidia PTX / AMD RDNA/CDNA is SIMD, but the execution is now MIMD. This pdf would then add a virtual machine on top of that, giving us an MIMD virtual-machine that compiles into SIMD instructions that's executed on MIMD hardware. I dunno where I'm going with this, I'm just amused by it, that's all. Each step makes sense but its kinda funny when you look at it all from the top down.

moonchild 4y ago
Of course GPUs are MIMD. They are also SIMD (or SIMT if you prefer). The same is true for CPUs. Parallelism is balanced along multiple axes, and if your problem is sufficiently heterogeneous, it seems completely plausible that the ideas in the linked paper would be useful for extracting additional parallelism from the hardware.
sitkack 4y ago
> So first off, GPUs are MIMD, in that each compute-unit / symmetric multiprocessor is independent of one another. This paper was from a SuperComputer 09, pretty sure the researchers are aware. Your comment would have great without the first line.
lostmsu 4y ago
Where can I read more on MIMD in 30xx?
- dragontamer 4y ago
  https://www.nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-architecture-whitepaper-v2.1.pdf https://www.nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-... Page 12. > In the Turing generation, each of the four SM processing blocks (also called partitions) had two primary datapaths, but only one of the two could process FP32 operations. The other datapath was limited to integer operations. GA10X includes FP32 processing on both datapaths, doubling the peak processing rate for FP32 operations. One datapath in each partition consists of 16 FP32 CUDA Cores capable of executing 16 FP32 operations per clock. Another datapath consists of both 16 FP32 CUDA Cores and 16 INT32 Cores, and is capable of executing either 16 FP32 operations OR 16 INT32 operations per clock. As a result of this new design, each GA10x SM partition is capable of executing either 32 FP32 operations per clock, or 16 FP32 and 16 INT32 operations per clock. All four SM partitions combined can execute 128 FP32 operations per clock, which is double the FP32 rate of the Turing SM, or 64 FP32 and 64 INT32 operations per clock.
  - lostmsu 4y ago
    As a non-involved, not sure I understand this quote correctly. Reads like Turing had 4 independent core equivalents, but 2 of them were incapable of FP; and Ampere has all 4 cores capable of doing either. But how are shaders scheduled to those cores? As a programmer, how do I see those cores and send code to run on them? Are there basic samples?
    - dragontamer 4y ago
      An NVidia 'SM' is the closest thing to a CPU core. Each SM has multiple MIMD capabilities as it is, but I was focusing upon the dual-issue pipelines of FP32+integer instructions simultaneously. > As a programmer, how do I see those cores and send code to run on them? When the following assembly instructions are send to the GPU, they will execute simultaneously add.s32 r1, r2 add.f32 r3, r4 That is: the NVidia SM can process a floating point (f32) instruction simultaneously with integer (s32) instructions. ------ It sounds like you are getting confused by a lot of other features in the GPU SM. I dunno where exactly, but it's a complicated beast. All I really think I can say is to study up on GPU microarchitecture for more details, there is a lot going on here and it'd be too difficult for me to cover it all in just a few comments.

moonchild 4y ago

Of course GPUs are MIMD. They are also SIMD (or SIMT if you prefer). The same is true for CPUs. Parallelism is balanced along multiple axes, and if your problem is sufficiently heterogeneous, it seems completely plausible that the ideas in the linked paper would be useful for extracting additional parallelism from the hardware.

sitkack 4y ago

> So first off, GPUs are MIMD, in that each compute-unit / symmetric multiprocessor is independent of one another. This paper was from a SuperComputer 09, pretty sure the researchers are aware. Your comment would have great without the first line.

lostmsu 4y ago

Where can I read more on MIMD in 30xx?

dragontamer 4y ago
https://www.nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-architecture-whitepaper-v2.1.pdf https://www.nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-... Page 12. > In the Turing generation, each of the four SM processing blocks (also called partitions) had two primary datapaths, but only one of the two could process FP32 operations. The other datapath was limited to integer operations. GA10X includes FP32 processing on both datapaths, doubling the peak processing rate for FP32 operations. One datapath in each partition consists of 16 FP32 CUDA Cores capable of executing 16 FP32 operations per clock. Another datapath consists of both 16 FP32 CUDA Cores and 16 INT32 Cores, and is capable of executing either 16 FP32 operations OR 16 INT32 operations per clock. As a result of this new design, each GA10x SM partition is capable of executing either 32 FP32 operations per clock, or 16 FP32 and 16 INT32 operations per clock. All four SM partitions combined can execute 128 FP32 operations per clock, which is double the FP32 rate of the Turing SM, or 64 FP32 and 64 INT32 operations per clock.
- lostmsu 4y ago
  As a non-involved, not sure I understand this quote correctly. Reads like Turing had 4 independent core equivalents, but 2 of them were incapable of FP; and Ampere has all 4 cores capable of doing either. But how are shaders scheduled to those cores? As a programmer, how do I see those cores and send code to run on them? Are there basic samples?
  - dragontamer 4y ago
    An NVidia 'SM' is the closest thing to a CPU core. Each SM has multiple MIMD capabilities as it is, but I was focusing upon the dual-issue pipelines of FP32+integer instructions simultaneously. > As a programmer, how do I see those cores and send code to run on them? When the following assembly instructions are send to the GPU, they will execute simultaneously add.s32 r1, r2 add.f32 r3, r4 That is: the NVidia SM can process a floating point (f32) instruction simultaneously with integer (s32) instructions. ------ It sounds like you are getting confused by a lot of other features in the GPU SM. I dunno where exactly, but it's a complicated beast. All I really think I can say is to study up on GPU microarchitecture for more details, there is a lot going on here and it'd be too difficult for me to cover it all in just a few comments.