5 ms·
The spreadsheet showing a bunch of different approaches: > Attempt Bandwidth Max Bandwidth Saturation Time to Code > Parallel STL 87 GB/s 204 GB/s 42.6
by inetknght 5y ago
The spreadsheet showing a bunch of different approaches:
> Attempt Bandwidth Max Bandwidth Saturation Time to Code
> Parallel STL 87 GB/s 204 GB/s 42.6% 1m
> Best CPU run 122 GB/s 204 GB/s 59.8% 60m
> Thrust 743 GB/s 936 GB/s 79.4% 1m
> Custom CUDA 817 GB/s 936 GB/s 87.3% 30m
> CUB 879 GB/s 936 GB/s 93.9% 5m
Then, benchmarks output:
> 8,115,337,000,378 instructions # 0.18 insn per cycle
# 4.71 stalled cycles per insn (83.33%)
> 1,820,092,697,347 branches # 168.291 M/sec (83.33%)
It's not clear to me which approach represents the output from the benchmarks. But that's a _lot_ of branches for the number of instructions executed. I suggest that might be a reason that the CPU didn't reach high RAM throughput.
- ashvardanian 5y agoThe numbers are for the last approach, committed to the public repo. I agree that it's a lot of branches, but expectedly so. Only 0.13% of all branches were missed, so the speculative execution almost always works. Even manual 4x unrolling didn't help.
- inetknght 5y agoIndeed? I wonder if you've tried different memory performance settings in your BIOS (which "might" invalidate your warranty) or different memory modules altogether
- ashvardanian 5y agoDidn't change in BIOS anything this time. But the entire system runs on liquid in a very cold, properly vented room. We must have cranked up the BIOS settings during the first boot.
- ashvardanian 5y agoAnd regarding the unrolling, I also removed the data dependency - accumulating into 4 different YMM registers. So it's most likely just the Infinity Fabric bottleneck, limiting our access to memory.
- 323 5y agoOn the other hand, high branch usage is the ideal scenario for using CPUs vs GPUs.
- dragontamer 5y agoNot necessarily. GPUs are outstandingly good at uniform branches. Only divergent branches are GPUs bad at.
- ashvardanian 5y agoI would further emphasize - locally uniform branches. Mostly the cores within the same warp should be well synchronized.
- dragontamer 5y agoIt also should be noted that CPUs are _ALSO_ bad at divergent branches. Its just that in the CPU-world, "divergent branches" is called "branch misprediction". In the GPU-world, we have a better idea of how the "branch predictor" (or the equivalent thing to the branch predictor) works. Its called SIMD-execution / branch divergence. ------- That being said: CPUs are way better than GPUs at divergent scenarios. Not only is the CPU branch predictor able to guess patterns, but its also able to execute in parallel to the rest of the CPU (out-of-order speculative execution and all). So in highly divergent branchy code, CPUs work but somewhat slowly (if the branch predictor can't predict, then its useless). But if there's a simple pattern, GPUs are great. If there's a pattern that can be detected at runtime, but is too difficult to program in to a GPU (ex: a binary search over a million elements will probably loop roughly 20 times), that's where CPUs win exceptionally over GPUs. The CPU-branch predictor will predict 20-loops in your binary search. It might be a little bit wrong (19-loops needed or 21-loops needed), but speeding up those 20-loops is a huge benefit.
- moonchild 5y agoWrt branches, GPUs want spatial locality, and CPUs want temporal locality :)