2 ms·
FWIW: your DRAM numbers are quoting clock speeds and not bandwidth. They aren't linear at all. In fact with enough cores you can easily saturate memory that w
by ajross 6y ago
FWIW: your DRAM numbers are quoting clock speeds and not bandwidth. They aren't linear at all. In fact with enough cores you can easily saturate memory that wide, and CPUs are getting wider just as fast as GPUs are. The giant Epyc AMD pushed out last fall has 8 (!) 64 bit DRAM channels, where IIRC the biggest NVIDIA part is still at 6.
- dragontamer 6y ago> 8 (!) 64 bit DRAM channels Yeah. And at 3200 Mbit/sec, that comes out to 200GB/s. (3200 MHz x 8-bytes (aka 64-bit) == 25GB/s. x8 channels == 200GB/s). > where IIRC the biggest NVIDIA part is still at 6. That's 6x *1024-bit* HBM2 channels. Total bandwidth is 2000GBps, or over 10x the speed of the "8x channel EPYC". Yeah, HBM2 is fat, extremely fat. ---------- *ONE* HBM2 channel offers over 300GBps bandwidth. And the A100 has *SIX* of them. Literally ONE HBM2 channel beats the speed of all 8x DDR4 EPYC memory channels working in parallel.
- ajross 6y agoYou're still quoting clock speeds. That's not how this works. Go check a timing diagram for a DRAM cycle in your part of choice and do the math.
- dragontamer 6y agoDo you know what 3200MHz / PC4-25600 DDR4 means? 25600 is the channel rate in (EDIT) MB/sec of the stick of RAM. That's 25GB/s for a 3200 MHz DDR4 stick. x8 (for 8-channels working in parallel) is 200GB/s. ----------- This has been measured in practice by Netflix: https://2019.eurobsdcon.org/slides/NUMA%20Optimizations%20in%20the%20FreeBSD%20Network%20Stack%20-%20Drew%20Gallatin.pdf https://2019.eurobsdcon.org/slides/NUMA%20Optimizations%20in... As you can see, Netflix's FreeBSD optimizations have allowed EPYC to reach 194GB/s measured performance (or just under the 200GB/s theoretical). And only with VERY careful NUMA-tuning and extreme optimizations were they able to get there.
- gbl08ma 6y agoAll of that is bandwidth and clock speed, not latency
- dragontamer 6y agoLook, if CPUs were better at memory latency, the BVH-traversal of raytracing would still be done on CPUs. BVH-tree traversals are done on the GPU now for a reason. GPUs are better at latency hiding and taking advantage of larger sets of bandwidth than CPUs. Yes, even on things like pointer-chasing through a BVH-tree for AABB bounds checking. GPUs have pushed latency down and latency-hiding up to unimaginable figures. In terms of absolute latency, you're right, GPUs are still higher latency than CPUs. But in terms of "practical" effects (once accounting for latency hiding tricks on the GPU, such as 8x way occupancy (similar to hyperthreading), as well as some dedicated datastructures / programming tricks (largely taking advantage of the millions of rays processed in parallel per frame), it turns out that you can convert many latency-bound problems into bandwidth-constrained problems. ----------- That's the funny thing about computer science. It turns out that with enough RAM and enough parallelism, you can convert ANY latency-bound problem into a bandwidth-bound problem. You just need enough cache to hold the results in the meantime, while you process other stuff in parallel. Raytracing is an excellent example of this form of latency hiding. Bouncing a ray off of your global data-structure of objects involved traversing pointers down the BVH tree. A ton of linked-list like current_node = current_node->next like operations (depending on which current_node->child the ray hit). From the perspective of any ray, it looks like its latency-bound. But from the perspective of processing 2.073 million rays across a 1920 x 1080 video game scene with realtime-raytracing enabled, its bandwidth bound.
- wmf 6y agoThat presentation shows 194 gigabits/s which is only ~24 gigabytes/s at the NIC; that requires ~96 gigabytes/s of memory bandwidth. Usable memory bandwidth on Milan is only <120 gigabytes/s which is about 60% of the theoretical max. DRAM never gets more than ~80% of theoretical max bandwidth because of command overhead (which is what I think ajross keeps alluding to). https://www.anandtech.com/show/16594/intel-3rd-gen-xeon-scalable-review/4 https://www.anandtech.com/show/16594/intel-3rd-gen-xeon-scal...
- mrb 6y agodragontamer is still correct. He quotes correct bandwidth numbers. EPYC's 8 channels of DDR4-3200 gets it to 204.8 GB/s (and, yes, that's bandwidth) Whereas Nvidia's A100 has over 2000 GB/s of memory bandwidth. That's 10-fold better.