4 ms·
GPUs simply have more bandwidth than CPUs. GPUs have huge latencies on VRAM, but are fed by GDDR6 or HBM2 at 1TBps at relatively cheap prices. There's no compar
by dragontamer 6y ago
GPUs simply have more bandwidth than CPUs. GPUs have huge latencies on VRAM, but are fed by GDDR6 or HBM2 at 1TBps at relatively cheap prices. There's no comparison with bandwidth.
NVidia NVLink is 300GBps GPU-to-GPU communications. In contrast, AMD EPYC is 50GBps CCX-to-CCX communications. Once again, there's no comparison with bandwidth, the GPUs simply win.
GPU-to-GPU is wider than CPU-to-CPU. VRAM-to-GPU is wider than RAM-to-CPU.
-------
If you have a problem that cares more about bandwidth than latency, then GPUs are simply the best architecture. Period. The pure compute performance is paired with absurdly huge bandwidth numbers.
Most consumer problems are latency-bound. HPC however, is specifically programmed to be bandwidth-bound instead. It takes effort and optimization to get there however.
- tachyonbeam 6y ago> Most consumer problems are latency-bound. Is that even true? I'd argue a lot of what happens on consumer devices can (or could) be done in parallel. There's some need for a fast response, but things like, responding to a button click and updating some internal state, take a trivial amount of computation to implement. Updating your UI in response to that button click, that's again something that can be parallelized.
- dragontamer 6y agoThink of the following line of code: blah = blah->next A simple linked-list traversal: 50-nanoseconds on a typical consumer CPU. This could represent a DOM-tree traversal. Or it could be a Javascript object being allocated. Or it could be the text getting added to the end of your rope-data structure. Or it could be a binary-search (finding a new address to check next). Or a malloc() (which traverses a linked-list to find memory). This dereference happens very often. 50-nanoseconds to hit main-RAM, 10-nanoseconds if L3 cache, 4-nanoseconds if in L2 cache, 1-nanosecond if in L1 cache. This sort of operation is very fast on CPUs, especially if you're cached. Heck, your CPU won't even wait on the result: its going to branch-predict the answer and start executing the next bit of code before it even knows its correct. ---------- That same memory-dereference is 300-nanoseconds from GPU VRAM, and maybe 100-nanoseconds if you're in GPU L1 cache. If you got a tree, its probably going to be far faster to traverse on a CPU. Where the GPU wins is Raytracing: you can have millions of individual tasks traversing the same tree, in parallel, at larger overall bandwidths than a CPU ever could. IE: GPUs can perform 1000 binary searches faster than a CPU can perform 1000 binary searches. But how many applications need 1000 binary searches? Usually, you need 1 binary search, and once you get the result, you need to do something else. (And that result can be branch predicted for even further accelerations to speed).
- jedbrown 6y agoI agree in general, but I'm not confident of this: > GPUs can perform 1000 binary searches faster than a CPU can perform 1000 binary searches. With several vector lanes and ILP on a 64+ core node, the 1000 parallel searches on the CPU may well beat the GPU. It'd have to be tested in any case.
- dragontamer 6y agoHmmmm... 1000 is still a "small" number when we're talking about GPU cores. I probably should have said 100,000 binary searches, or 1,000,000 binary searches. You're right, 1000 is a small enough number where the CPU might still win. If its a large-enough data-structure to hit VRAM (ie: a binary search across a 8GB vector), that's what'd be best for the GPU. Raytracing is millions rays per scene. 4k is 8,294,400 rays at 1-sample per pixel, and each one of those rays has to traverse a BVH tree (similar to a binary tree). Each of those rays MIGHT set off a 2nd or 3rd ray, depending on what its bouncing against. The big BVH tree that organizes all of the video game triangles / objects is shared for all rays. That's the "1000s of binary tree searches in parallel" I was trying to reference. But I guess its closer to 10s of millions.
- fulafel 6y ago"Could be done in parallel" is a very different space. As is painfully evident, sw inertia/ path dependency is huge and we get widespread big changes on timescale of decades.
- jedbrown 6y agoYou're arguing at a particular granularity and even that doesn't hold uniformly. A64FX is HBM2 at equivalent bandwidth to GPUs (with lower power). CCX is much finer granularity than an entire GPU, so not a direct comparison. L3 bandwidth on EPYC is multi-TB/s. Fat GPU nodes can readily overload network interfaces so if bisection bandwidth is your concern, CPU nodes are good. > HPC however, is specifically programmed to be bandwidth-bound instead. This is wishful thinking. Lots of applications used to justify the US exascale program (and others) are latency-bound. Climate, weather, unsteady CFD, and much of mesoscale materials science and molecular dynamics are run at their latency limit in most scientific studies (one-off scaling studies notwithstanding). There's an unfortunate disconnect between what scientific computing actually needs versus what funders and the media portray.
- dragontamer 6y ago> A64FX is HBM2 at equivalent bandwidth to GPUs (with lower power). CCX is much finer granularity than an entire GPU, so not a direct comparison. But only when using SVE512 SIMD-units, which are grossly similar to GPU SIMD units. At a minimum, SIMD reigns supreme. Even Intel only gets its max bandwidth when using AVX512 units. Once you start rewriting your inner loops to run with SIMD, its not too difficult to start thinking about a dedicated SIMD-accelerator, or GPU, to do the job. > L3 bandwidth on EPYC is multi-TB/s. 32-bytes / infinity fabric cycle. 16-CCX per EPYC chip. 3GHz == 1.5 TBps. I dunno about "multi-TB/s", but its over 1 TBps... sure. But only if used in parallel. EPYC only has 32MBs of L3 per CCX. To achieve the full bandwidth, you need to split the problem into each CCX (which asserts a MESI-like "exclusive" lock when writing to a L3 location: preventing other L3 caches from reading-or-writing there). Even then, EPYC's L3 cache is small compared to GPU-VRAM. Radeon VII 1TBps VRAM applies at full speed with atomics / synchronizations (in fact, the atomics / synchronization to L2 cache. I just don't have L2 numbers for that GPU...). I would expect Radeon VII's L2 cache to have more bandwidth than EPYC's L3 cache (Indeed: the Radeon VII L2 cache is in front of a 1TBps HBM2 cluster). If we traverse up the GPU cache structure, you get 10TBps+ on __shared__ or LDS RAM, which is used as atomic-synchronization points or thread-barriers within a workgroup (a batch of up to 1024 cudaThreads). As such: synchronization between threads (within a large workgroup), or even across the device, is reasonably efficient. The downside of this comparison is that 1024 cudaThreads only have access to 64kB of __shared__ RAM. So it isn't really comparable from a size perspective. ------ Your "TBps" estimate on EPYC's L3 cache however, is misleading. Because you spend significant amounts of MESI messages passing cache lines back and forth between CCX to get there. Compared to a unified L2 on the GPU (or unified VRAM), its just not really comparable. EPYC L3 is somewhere between GPU L2 and GPU __shared__, in terms of memory hierarchy and complexity of use.