3 ms·

I'm probably missing something but why not use gpus for parallel processing?

by pier25 2y ago

JackSlateur 2y ago
Because GPU are physically built to manage parallel task, but only a few kinds They are very specialized CPU are generics, they have lots of transistors to handle a lot of different instructions
- Groxx 2y ago
  Also moving data to and from the GPU takes MUCH more time than between CPU cores (though combined chips drastically lower this difference).
  - markhahn 2y ago
    in the olden days of gp-cpu this was certainly true. is it still true? do you just mean "latency overhead for setting up a single PCIe transaction is much larger than flinging a cache line across QPI/etc"?
- markhahn 2y ago
  I think this is a really unfortunate way to explain it. The issue is not that CPUs have a lot of different instructions - hardly anyone uses decimal math instructions, for instance, and no one cares about baroque complex addressing modes. The difference is that GPU code is designed to tolerate latency by having lots of loop iterations treated as threads. A modern CPU tolerates latency by maintaining the readiness of hundreds of individual instructions ("in flight") - essentially focusing on minimizing the execution latency of each instruction. (which also explains how CPUs use caches and very high clocks, but wind up with somewhat fewer cores and threads.) (note that I'm using cores and threads correctly here, not the nvidia way.)
nine_k 2y ago
GPUs work on massive amounts of data in parallel, but they execute basically the same operations every step, maybe skipping or slightly varying some steps depending on the data seen by a particular processing unit. But processing units cannot execute independent streams of instructions. GPUs of course have several parts that can work in parallel, but they are few, and every part consists of large amounts that execute the same instruction stream simultaneously over a large chunk of data.
- winwang 2y ago
  This is not true. Take the NVidia 4090. 128 SMs = 4x128=512 SMSPs. This is the number of warps which can execute independently of each other. In contrast, a warp is a 32-width vector, i.e. 32 "same operations", and up to 512 different batches in parallel. So, it's more like a 512-core 1024-bit vector processor. That being said, I believe the typical number of warps to saturate an SM is normally around 6 rather than 4, so more like 768 concurrent 32-wide "different" operations to saturate compute. Of course, the issue with that is you get into overhead problems and memory bandwidth issues, both of which are highly difficult to navigate around -- the register file storing all the register of each process is extremely power-hungry (in fact, the most power-hungry part I believe), for example. A PPU with less vector width (e.g. AVX512) would have proportionally more overhead (possibly more than linearly so in terms of the circuit design). This is without talking about how most programs depend on latency-optimized RAM (rather than bandwidth-optmized GDDR/HBM).
  - nine_k 2y ago
    I'm happy to stand corrected; apparently my idea about GPUs turned obsolete by now.
  - faangguyindia 2y ago
    The Nvidia 4090 indeed has 128 SMs, but the formula you provided (128 SMs = 4x128=512 SMSPs) isn't quite accurate. Each SM contains 64 CUDA cores (not SMSPs), and these are the units responsible for executing the instructions from different warps. The term "SMSP" isn't typically used to describe CUDA cores or warps in Nvidia’s architecture.
    - rnrn 2y ago
      winwang’s comment is correct, yours is wrong. “cuda core” refers to one lane within the SIMT/SIMD ALUs. These lanes within a SMSP don’t execute independently. The term SMSP is definitely used for nvidia’s architecture : https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html#metrics-decoder https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.... > smsp > Each SM is partitioned into four processing blocks, called SM sub partitions. The SM sub partitions are the primary processing elements on the SM. (Note that this kernel profiling guide doesn’t use the term “cuda cores” at all) Also there are 128 “cuda cores” per SM in 4090, not 64 : https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvidia-ada-gpu-architecture.pdf https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvid... > Each SM in AD10x GPUs contain 128 CUDA cores

I'm probably missing something but why not use gpus for parallel processing?

by pier25 2y ago

I'm probably missing something but why not use gpus for parallel processing?

Because GPU are physically built to manage parallel task, but only a few kinds They are very specialized CPU are generics, they have lots of transistors to handle a lot of different instructions

Groxx 2y ago
Also moving data to and from the GPU takes MUCH more time than between CPU cores (though combined chips drastically lower this difference).
- markhahn 2y ago
  in the olden days of gp-cpu this was certainly true. is it still true? do you just mean "latency overhead for setting up a single PCIe transaction is much larger than flinging a cache line across QPI/etc"?
markhahn 2y ago
I think this is a really unfortunate way to explain it. The issue is not that CPUs have a lot of different instructions - hardly anyone uses decimal math instructions, for instance, and no one cares about baroque complex addressing modes. The difference is that GPU code is designed to tolerate latency by having lots of loop iterations treated as threads. A modern CPU tolerates latency by maintaining the readiness of hundreds of individual instructions ("in flight") - essentially focusing on minimizing the execution latency of each instruction. (which also explains how CPUs use caches and very high clocks, but wind up with somewhat fewer cores and threads.) (note that I'm using cores and threads correctly here, not the nvidia way.)

nine_k 2y ago

GPUs work on massive amounts of data in parallel, but they execute basically the same operations every step, maybe skipping or slightly varying some steps depending on the data seen by a particular processing unit. But processing units cannot execute independent streams of instructions. GPUs of course have several parts that can work in parallel, but they are few, and every part consists of large amounts that execute the same instruction stream simultaneously over a large chunk of data.

winwang 2y ago
This is not true. Take the NVidia 4090. 128 SMs = 4x128=512 SMSPs. This is the number of warps which can execute independently of each other. In contrast, a warp is a 32-width vector, i.e. 32 "same operations", and up to 512 different batches in parallel. So, it's more like a 512-core 1024-bit vector processor. That being said, I believe the typical number of warps to saturate an SM is normally around 6 rather than 4, so more like 768 concurrent 32-wide "different" operations to saturate compute. Of course, the issue with that is you get into overhead problems and memory bandwidth issues, both of which are highly difficult to navigate around -- the register file storing all the register of each process is extremely power-hungry (in fact, the most power-hungry part I believe), for example. A PPU with less vector width (e.g. AVX512) would have proportionally more overhead (possibly more than linearly so in terms of the circuit design). This is without talking about how most programs depend on latency-optimized RAM (rather than bandwidth-optmized GDDR/HBM).
- nine_k 2y ago
  I'm happy to stand corrected; apparently my idea about GPUs turned obsolete by now.
- faangguyindia 2y ago
  The Nvidia 4090 indeed has 128 SMs, but the formula you provided (128 SMs = 4x128=512 SMSPs) isn't quite accurate. Each SM contains 64 CUDA cores (not SMSPs), and these are the units responsible for executing the instructions from different warps. The term "SMSP" isn't typically used to describe CUDA cores or warps in Nvidia’s architecture.
  - rnrn 2y ago
    winwang’s comment is correct, yours is wrong. “cuda core” refers to one lane within the SIMT/SIMD ALUs. These lanes within a SMSP don’t execute independently. The term SMSP is definitely used for nvidia’s architecture : https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html#metrics-decoder https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.... > smsp > Each SM is partitioned into four processing blocks, called SM sub partitions. The SM sub partitions are the primary processing elements on the SM. (Note that this kernel profiling guide doesn’t use the term “cuda cores” at all) Also there are 128 “cuda cores” per SM in 4090, not 64 : https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvidia-ada-gpu-architecture.pdf https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvid... > Each SM in AD10x GPUs contain 128 CUDA cores