5 ms·

- Knock Knock * Who's there? - GPUs. * Closes door. - Knock knock - Knock knock - Do you really want to ignore ~90% of the compute throughput and ~90% of

- Knock Knock * Who's there? - GPUs. * Closes door. - Knock knock - Knock knock - Do you really want to ignore ~90% of the compute throughput and ~90% of the memory bandwidth of your machine? - Knock knock - ... ---- An AMD EPYC 7742 (Milan, Zen 3, announced last week, high-end server model) has ~100 GB/s DRAM bandwidth and 1-2 TFLOPS double precision. A single PCI-e A100 has ~2Tb/s DRAM bandwidth, and 10-20 TFLOPs double precision. You can pack 4x of these per CPU socket... so you end up at 8 TB/s and 40-80 TFLOPs... That makes the GPUs like 99% of both the compute and memory BW of a socket.

dragontamer 6y ago
Well, CPUs vs GPUs is pretty interesting actually. The #1 supercomputer right now is CPU-only, the A64Fx by Fujitsu. But you're right in that GPU-parallelism is the obvious parallel architecture available to modern programmers. For anyone spending less than $500,000 on a computer, GPUs are probably the most compute for the least amount of cost. --------- The issue is that there's two difference concepts here: 1. Parallelism -- When a problem you're working on has an undefined ordering to it explicitly. Protocols and communications for example have undefined orders. You could write a state-machine explicitly, but writing a thread is usually much simpler to understand. Ultimately: this is cofunctions, Python Yield / generators, async-code, etc. etc. Remember: Djikstra invented semaphores on a single-core processor. The concept of "single core parallelism" is hugely useful and can grossly simplify code if used correctly. 2. High performance -- This is a different concept entirely!! GPUs are within the realm of high-performance parallelism (and not just merely "conceptual parallelism" in #1). GPUs are hard because anyone who uses a GPU expects high performance. Be it the video gamer who wants raytracing on his video game, or the machine learning enthusiast training a billion neurons. Writing a simple "GPU Hello World" is easy as all heck actually. Its all the performance stuff that gets really hard.
- sudosysgen 6y ago
  The A64Fx really is something between a CPU and a GPU, if we're being fair.
  - dragontamer 6y ago
    The only GPU-like thing it has is on-board HBM2 RAM (which other CPUs could do, but haven't bothered to do for some reason. Except that one Xeon Phi using the HMC competitor to HBM). The SIMD-model of A64Fx: the SVE instruction set of ARM, is coming to mainstream Neoverse V-series ARM chips within a year or two. SVE is CPU-parallelism. Well, its SIMD, but its just 512-bit SIMD (equivalent to AVX512 in width). So that's smaller scale SIMD / CPU parallelism IMO. Those A64Fx are going to be branch-predicting and executing those SIMD-instructions out-of-order. Its a CPU. ------- I consider GPUs to be the simpler in-order but more parallel stuff (AMD, NVidia, Xilinx maybe). But you're right in that the line keeps getting blurred each year. (NVidia's dual-issue pipelines is a very CPU-like feature. AVX512 and SVE pushing 512-bits is getting closer-and-closer to GPU-like SIMD widths)
    - sudosysgen 6y ago
      GPUs can now basically do out-of-order execution using dynamic warp formation and driver-based prediction and optimization. Branch prediction is useless in massively parallel workflows, because of linearization. GPU SIMD is also more complicated than full-SIMD. Nowadays with work-stealing and multiple workgroups, you can have very similar performance characteristics to parallel SIMD on CPUs, just with more perfromance. And in any case, OOE for SIMD instructions loses a lot of it's usefulness on many workflows.
      - dragontamer 6y ago
        > GPUs can now basically do out-of-order execution using dynamic warp formation and driver-based prediction and optimization. That's... not OOE. If you have: add [eax], ebx add ecx, edx A CPU can execute "add ecx, edx" before the "add [eax], ebx" (especially if the [eax] memory read/write is far away: like in DDR4 or worse). A GPU will ALWAYS execute the 1st line before the 2nd line. Dynamic warp formation means that SIMD-lanes can split-up in your warp, but none of the warp is executing anything out-of-order. > Branch prediction is useless in massively parallel workflows, because of linearization. Stockfish NNUE argues otherwise. Only part of the network updates at any given time. Its a massively-parallel problem, but you need dynamic / branch prediction to see which parts of the Neural-Net are actually updating. The idea of CPU-designed networks (Neural Nets that PARTIALLY update, instead of a GPU-network that updates entirely) is intriguing to me. There might be other situations where branch prediction is helpful in other largely parallel problems. ---------- > And in any case, OOE for SIMD instructions loses a lot of it's usefulness on many workflows. More like the SIMD offers enough parallelism to saturate the machine. But NNUE really shows where OOE / branch prediction can suddenly become unexpectedly helpful. Classic SIMD algorithms do not account for branch prediction / OOE benefits. The very existence of high-speed SIMD + OOE + branch prediction is an unexplored part of computer science IMO.
deleted 6y ago
[deleted]
imtringued 6y ago
GPU nodes are extremely expensive. People aren't renting them by accident.

dragontamer 6y ago

Well, CPUs vs GPUs is pretty interesting actually. The #1 supercomputer right now is CPU-only, the A64Fx by Fujitsu. But you're right in that GPU-parallelism is the obvious parallel architecture available to modern programmers. For anyone spending less than $500,000 on a computer, GPUs are probably the most compute for the least amount of cost. --------- The issue is that there's two difference concepts here: 1. Parallelism -- When a problem you're working on has an undefined ordering to it explicitly. Protocols and communications for example have undefined orders. You could write a state-machine explicitly, but writing a thread is usually much simpler to understand. Ultimately: this is cofunctions, Python Yield / generators, async-code, etc. etc. Remember: Djikstra invented semaphores on a single-core processor. The concept of "single core parallelism" is hugely useful and can grossly simplify code if used correctly. 2. High performance -- This is a different concept entirely!! GPUs are within the realm of high-performance parallelism (and not just merely "conceptual parallelism" in #1). GPUs are hard because anyone who uses a GPU expects high performance. Be it the video gamer who wants raytracing on his video game, or the machine learning enthusiast training a billion neurons. Writing a simple "GPU Hello World" is easy as all heck actually. Its all the performance stuff that gets really hard.

sudosysgen 6y ago
The A64Fx really is something between a CPU and a GPU, if we're being fair.
- dragontamer 6y ago
  The only GPU-like thing it has is on-board HBM2 RAM (which other CPUs could do, but haven't bothered to do for some reason. Except that one Xeon Phi using the HMC competitor to HBM). The SIMD-model of A64Fx: the SVE instruction set of ARM, is coming to mainstream Neoverse V-series ARM chips within a year or two. SVE is CPU-parallelism. Well, its SIMD, but its just 512-bit SIMD (equivalent to AVX512 in width). So that's smaller scale SIMD / CPU parallelism IMO. Those A64Fx are going to be branch-predicting and executing those SIMD-instructions out-of-order. Its a CPU. ------- I consider GPUs to be the simpler in-order but more parallel stuff (AMD, NVidia, Xilinx maybe). But you're right in that the line keeps getting blurred each year. (NVidia's dual-issue pipelines is a very CPU-like feature. AVX512 and SVE pushing 512-bits is getting closer-and-closer to GPU-like SIMD widths)
  - sudosysgen 6y ago
    GPUs can now basically do out-of-order execution using dynamic warp formation and driver-based prediction and optimization. Branch prediction is useless in massively parallel workflows, because of linearization. GPU SIMD is also more complicated than full-SIMD. Nowadays with work-stealing and multiple workgroups, you can have very similar performance characteristics to parallel SIMD on CPUs, just with more perfromance. And in any case, OOE for SIMD instructions loses a lot of it's usefulness on many workflows.
    - dragontamer 6y ago
      > GPUs can now basically do out-of-order execution using dynamic warp formation and driver-based prediction and optimization. That's... not OOE. If you have: add [eax], ebx add ecx, edx A CPU can execute "add ecx, edx" before the "add [eax], ebx" (especially if the [eax] memory read/write is far away: like in DDR4 or worse). A GPU will ALWAYS execute the 1st line before the 2nd line. Dynamic warp formation means that SIMD-lanes can split-up in your warp, but none of the warp is executing anything out-of-order. > Branch prediction is useless in massively parallel workflows, because of linearization. Stockfish NNUE argues otherwise. Only part of the network updates at any given time. Its a massively-parallel problem, but you need dynamic / branch prediction to see which parts of the Neural-Net are actually updating. The idea of CPU-designed networks (Neural Nets that PARTIALLY update, instead of a GPU-network that updates entirely) is intriguing to me. There might be other situations where branch prediction is helpful in other largely parallel problems. ---------- > And in any case, OOE for SIMD instructions loses a lot of it's usefulness on many workflows. More like the SIMD offers enough parallelism to saturate the machine. But NNUE really shows where OOE / branch prediction can suddenly become unexpectedly helpful. Classic SIMD algorithms do not account for branch prediction / OOE benefits. The very existence of high-speed SIMD + OOE + branch prediction is an unexplored part of computer science IMO.

deleted 6y ago

[deleted]

imtringued 6y ago

GPU nodes are extremely expensive. People aren't renting them by accident.