7 ms·
Failing to reach DDR4 bandwidth
- inetknght 5y agoThe spreadsheet showing a bunch of different approaches: > Attempt Bandwidth Max Bandwidth Saturation Time to Code > Parallel STL 87 GB/s 204 GB/s 42.6% 1m > Best CPU run 122 GB/s 204 GB/s 59.8% 60m > Thrust 743 GB/s 936 GB/s 79.4% 1m > Custom CUDA 817 GB/s 936 GB/s 87.3% 30m > CUB 879 GB/s 936 GB/s 93.9% 5m Then, benchmarks output: > 8,115,337,000,378 instructions # 0.18 insn per cycle # 4.71 stalled cycles per insn (83.33%) > 1,820,092,697,347 branches # 168.291 M/sec (83.33%) It's not clear to me which approach represents the output from the benchmarks. But that's a _lot_ of branches for the number of instructions executed. I suggest that might be a reason that the CPU didn't reach high RAM throughput.
- ashvardanian 5y agoThe numbers are for the last approach, committed to the public repo. I agree that it's a lot of branches, but expectedly so. Only 0.13% of all branches were missed, so the speculative execution almost always works. Even manual 4x unrolling didn't help.
- inetknght 5y agoIndeed? I wonder if you've tried different memory performance settings in your BIOS (which "might" invalidate your warranty) or different memory modules altogether
- ashvardanian 5y agoDidn't change in BIOS anything this time. But the entire system runs on liquid in a very cold, properly vented room. We must have cranked up the BIOS settings during the first boot.
- ashvardanian 5y agoAnd regarding the unrolling, I also removed the data dependency - accumulating into 4 different YMM registers. So it's most likely just the Infinity Fabric bottleneck, limiting our access to memory.
- 323 5y agoOn the other hand, high branch usage is the ideal scenario for using CPUs vs GPUs.
- dragontamer 5y agoNot necessarily. GPUs are outstandingly good at uniform branches. Only divergent branches are GPUs bad at.
- ashvardanian 5y agoI would further emphasize - locally uniform branches. Mostly the cores within the same warp should be well synchronized.
- dragontamer 5y agoIt also should be noted that CPUs are _ALSO_ bad at divergent branches. Its just that in the CPU-world, "divergent branches" is called "branch misprediction". In the GPU-world, we have a better idea of how the "branch predictor" (or the equivalent thing to the branch predictor) works. Its called SIMD-execution / branch divergence. ------- That being said: CPUs are way better than GPUs at divergent scenarios. Not only is the CPU branch predictor able to guess patterns, but its also able to execute in parallel to the rest of the CPU (out-of-order speculative execution and all). So in highly divergent branchy code, CPUs work but somewhat slowly (if the branch predictor can't predict, then its useless). But if there's a simple pattern, GPUs are great. If there's a pattern that can be detected at runtime, but is too difficult to program in to a GPU (ex: a binary search over a million elements will probably loop roughly 20 times), that's where CPUs win exceptionally over GPUs. The CPU-branch predictor will predict 20-loops in your binary search. It might be a little bit wrong (19-loops needed or 21-loops needed), but speeding up those 20-loops is a huge benefit.
- moonchild 5y agoWrt branches, GPUs want spatial locality, and CPUs want temporal locality :)
- jhokanson 5y agoIt is not exactly clear to me what is going on with threads (I guess you are using all of them?). I haven't done too much in this space but anecdotally I've had better luck if my summation is explicitly split into sub-summation tasks. It is not clear if that is being done here. It looks like a single summation loop that the author is expecting the computer to magically split across multiple threads. I'd be interested in seeing what this looks like if instead the task were to add chunks of the original dataset into results per thread (e.g, first 8000 samples on first thread, next 8000 on 2nd thread, etc.), with a final accumulation loop across all threads. Again, the author may be trying this and this is not my area of expertise but I've had decent luck saturating the memory bus with a similar approach.
- ashvardanian 5y agoHere is the source and the threads: https://github.com/unum-cloud/ParallelReductions/blob/fd16d9f64d18a53fa662b636359a89b579bdd2b3/reduce_cpu.hpp#L254 https://github.com/unum-cloud/ParallelReductions/blob/fd16d9... OFC we don't expect the compiler to instantiate them for us, it's not OpenMP :) That one we covered in previous articles. OpenMP gave us about 50 GB/s with all cores enabled and 80 GB/s with part of them disabled.
- jhokanson 5y agoMy c++ is not great (so it is hard for me to tell what is going on) and I'm used to OpenMP where my understanding has always been that you tend to get a single thread per processor (or per hyper-thread) -- not sure if that is guaranteed with the way your code is laid out? Perhaps it really is a NUMA issue as others suggest. I will note that one other variation I had (as it looks like you are already splitting across threads) is that the chunk sizes were actually smaller than the # of threads which meant a faster thread would take more chunks rather than waiting on the slowest thread. Good luck!
- a_t48 5y agoIs there an advantage to using taskflow for parallel for, if you already have another threadpool implementation? I recently removed taskflow in a project that was only being used for a parallel for loop (as part of a larger refactor, the code had a number of issues...), and I'm wondering if that was a mistake now that I see that pattern somewhere else. :)
- dragontamer 5y agoGPUs are in fact easier to reach high performance these days IMO. Sure, you gotta learn how to stride memory and possibly do it across workgroups, but those GPUs really have patterns that easily hit their specs (be it FLOPs or Memory Bandwidth) GPUs are dumb, but that makes them kinda simpler for these kinds of simple tests. CPUs, in particular the Thread ripper Pro, have very complex memory hierarchies. Unless they experiment with NUMA mode (4 or 8 nodes maybe), I wouldn't expect much improvement. To max out the memory Bandwidth on a CPU, you really gotta understand MESI / cache snooping / false sharing, split up your accesses across the chip let's (easier to do with NUMA turned on) and code in a NUMA aware manner. --------- I'm not saying GPUs are always better. But this particular test is really bloody simple and ideal for GPUs.
- ashvardanian 5y agoYeah, completely agree. Sadly we live in the convergence era. Now GPUs are also multi-chip (starting with MI200) and the latencies will likely become unpredictable once again :)
- tuetuopay 5y agoYou are missing one critical element: NUMA awareness. Because of the Chiplet design, Threadrippers and EPYC chips are basically NUMA systems, like you would with a multi-socket motherboard. This has an impact on both memory latency and bandwidth. Redo your test, but pin each thread to a CPU thread, and its memory pool to a stick of ram attached directly to the CCX of said thread. The bandwidth you are measuring is the Infinity Fabric's bandwidth, NOT the DDR4 bandwidth. You can read more about infinity fabric and its bandwidth here: https://en.wikichip.org/wiki/amd/infinity_fabric https://en.wikichip.org/wiki/amd/infinity_fabric
- ashvardanian 5y agoThanks, that's a valid suggestion! But in that case I would be forced to split my array into smaller chunks in different disjoint region of address space, right?
- vardump 5y agoCame to say exactly this. A lot of code is wasting a ton of performance on servers because of not being NUMA aware. We should also test high throughput/performance code on NUMA systems. Remember to allocate local memory for your threads — IOW, be NUMA aware. Going through interconnect(s) is slow.
- jjoonathan 5y agoHow is that generally done? Platform specific libraries? Utilities that ask the OS to pin a program to a node?
- pkaye 5y agoIn Linux you can use numactl command to pin a process.
- cyber_kinetist 5y agolibnuma probably does the job. https://github.com/numactl/numactl https://github.com/numactl/numactl
- StillBored 5y agoI would try something like this on gravaton2/3 or a recent POWER machine, in my experience its a lot easier to get reasonable memory perf out of !intel machines.
- bee_rider 5y agoThe threadripper is also a !intel chip.
- ashvardanian 5y agoThreadripper is by AMD
- bee_rider 5y agoYeah that was what I was pointing out -- I was mirroring the previous, who I think meant !intel as "not intel."
- tryp 5y agoSome folks are just not TradeMarksters. They'll teach you that Intel makes processors that implement x86 and amd64 instruction sets. You can Google things on DuckDuckGo. They'll blow their nose on the Kleenex that they bought from GenericPaperProductsCo. At some restaurants in the US South you'll have to let your server know what kind of Coke you want: cola, lemon-lime, or the Doctor one.
- deleted 5y ago[deleted]
- jhokanson 5y agoDoes AMD have folks that you can reach out to regarding this? I know Intel has MKL and all the work around its own compiler for maximum speed. This seems like it should be trivial for someone at AMD to put together as an example of how to do things like this correctly ...
- deleted 5y ago[deleted]
- ggreg84 5y agoI bet AMD itself is not able to get even close to the 204 GB/s bandwidth that they "promise" for these CPUs. AMD instructions for measuring peak bandwidth are here: https://developer.amd.com/spack/stream-benchmark/ https://developer.amd.com/spack/stream-benchmark/ (essentially does a multi threaded memcpy). The script there will pin 1 thread per CCD, and with 8 threads you'll get close to peak bandwidth on CPUs with 8 CCDs (adding more threads doesn't help). I've run it in the same CPUs that AMD uses there, following their instructions to the letter, which should be able to achieve 204 GB/s, but the best I've seen is about 165 GB/s. That's 80% of what AMD advertises. HOWEVER, THIS NUMBER IS A LIE. It is using the AMD compiler with a flag called "-fnt-store" to turn all stores into non-temporal stores. This is something that no real-world application in practice would actually do because it does not make sense doing in general. Without this flag, the peak bandwidth that AMD scripts report is about 124 GB/s. That's 60% of what AMD advertises and the best that most applications can aim to obtain in practice (the blog post author is really close to it). The fine print (https://www.amd.com/en/products/cpu/amd-epyc-7742 https://www.amd.com/en/products/cpu/amd-epyc-7742) does say that 204 GB/s is a "theoretical" number. So I bought a "theoretical" 204 GB/s CPU that delivers 60% of what it "theoretically" promised. I feel scammed. --- Instead, if I run STREAM on my GPU, I get close to 100% of what the GPU promises, and about 90% or so of it in practice.
- ashvardanian 5y agoThose are great links, thanks for sharing! My memcpy-s were even slower :)
- sliken 5y agoYou seem surprised, this is pretty much the rule for ALL CPUs. The 204GB/sec number is simply DDR4-3200 x 64 bits (width of one memory channel) x 8 (number of memory channels = 204,800MB/sec or "roughly" 204GB/sec (assuming GB = 10^9 not 2^30). This is called "peak" bandwidth, as is a never to exceed number. 60-80% of peak is common with tuned code, and less without. Sure GPUs, which are all about bandwidth do better. Much of a GPU design is about throughput and hiding latency with multiple requests/threads/whatever. CPUs have different design goals and worry much more about latency and often have to worry about NUMA/memory coherency. Not to mention in the general case often you have one cache miss per core, which isn't necessarily enough to saturate the memory bus. Try apple M1, other ARMs, Alpha, PA-risc, Sparc, Intel, etc and you'll find the same story.