6 ms·

My understanding of conventional wisdom from maybe 15 years ago was that we'd run out of memory bandwidth for high core-count processors. Is this now a factor?

My understanding of conventional wisdom from maybe 15 years ago was that we'd run out of memory bandwidth for high core-count processors. Is this now a factor? I know memory has gotten faster, and multi-channel of course helps, but is that all really enough for typical workloads?

cma 7y ago
It has a separate dedicated IO die, allowing memory access/latency to all cores. I think threadripper does have less bandwidth than the 64 core Epyc part though.
- amarshall 7y ago
  Indeed as TR is 4-channel and Epyc is 8-channel. If one needs ECC, TR is a bit more limited still due to unbuffered ECC memory having lower max speeds presently.
  - cma 7y ago
    Can't edit now but I meant to write "allowing [uniform] memory access/latency to all cores"
greglindahl 7y ago
CPUs keep on getting more memory channels per socket. That trend was already visible 15 years ago, when AMD moved the northbridge onto the CPU. Scaling it depends on scaling pin counts, which are now amazing.
- wtallis 7y ago
  To put some numbers to this: In 2005, Opterons used a 940-pin socket and had dual-channel DDR, for 6.4GB/s total bandwidth to feed two cores. In 2009, Xeons the LGA1366 socket to provide triple-channel DDR3, 32GB/s for four (later six) cores. Now, we're up to 4094 pin sockets providing 8 channels of DDR4 for a total of 204.8GB/s of memory bandwidth for 64 cores, which is exactly the same per-core bandwidth we had in 2005. (And for comparison, a $200 graphics card will also have around 200GB/s of memory bandwidth.)
  - vbezhenar 7y ago
    Unrelated question, what about RAM latency? Did it significantly improve since 2005?
    - dwrodri 7y ago
      So at the consumer level, DDR2[popular standard c. 2005] could be clocked up to 266 MHz whereas DDR4[popular now] clocks in at 1600MHz. So a little more than a 6x speedup in transaction times. Unlike processors, whose features make it quite difficult to compare against each other, DRAM has stayed relatively straightforward: its still just a bunch of densely packed flip-flops connected to a memory controller that offers bank-strided column/row access. As you can imagine, processor performance has grown a lot more than 6x over the years. The amount of cores in a package alone has increased more than 6x! So how do we get around this? 1) Larger caches: the most efficient optimization to any slow task is to simply not perform said task. Having a larger L3 cache means sending requesting fewer pages from DRAM. 2) “Hiding” latency with bandwidth: if I have to incur a (relatively) large time penalty tp go to DRAM, I better make sure to make it worth my time. Instead of always getting data in 4KiB pages, often times modern processors support “superpages” all the way up to the gigabytes. If you’re going to go drive out to the store, it makes sense to buy groceries for the week instead of just the candy bar you’re craving right now, right? That way when you actually have to cook the work has already been done. I am probably missing some nuance here, especially when it comes to memory controllers and putting caches on DRAM, but this is the gist.
      - vbezhenar 7y ago
        But frequency of RAM and latency is not the same. DDR2-800 can have something like 4-4-4-12 timings, while DDR4-2666 can have something like 15-17-17-35 timings. So their result latency will be around the same! And those numbers I just googled, I did not made them up.
      - Retric 7y ago
        DDR2 vs DDR4 has effectively zero impact on latency. Due to the physical distances between RAM modules and the CPU, that’s been stuck within a narrow range for decades. https://en.wikipedia.org/wiki/CAS_latency https://en.wikipedia.org/wiki/CAS_latency The specific implication details make a difference, but nothing is going to provide a 4x improvement in latency so other considerations take precedence. The real change is ever growing cache sizes allowing CPU’s to minimize random RAM reads.
codeflo 7y ago
I’m interested in this as well. As opposed to CPUs from 15 years ago, the 3990X has 256 MB L3 cache and around 100 GB/s of memory bandwidth, which should help a bit. What kinds of workloads can we expect to be bandwidth-botllenecked still?
- BubRoss 7y ago
  If a program is structured in a particular way to run through large spans of contiguous memory but only do trivial operations, then memory bandwidth can be maxed out by the prefetcher and the cache wouldn't be used. Things like adding two huge images with SIMD and writing out a new image could be like this. It would be much better of course if the image is tiled and more operations are done per tile, which would use the cache. Good use of the prefetcher but poor use of the cache could still run through memory bandwdith.
imtringued 7y ago
The rule of thumb is 25GB/s per channel. At 8 channels you have 200GB/s. Unless you are working with SIMD it is unlikely that you actually hit these numbers.
- blattimwind 7y ago
  Actually it's super easy, barely an inconvenience, since the raw memory bandwidth you get per core is around 1 byte per cycle and half that per thread. A 3900X with 12 cores / 24 threads getting a total memory bandwidth of 57.6 GB/s (using DDR4-3600) has just 0.6 bytes / cycle and thread. Meanwhile the CPU can actually achieve ALU throughputs that are 20 times greater than that. Even encryption algorithms are much faster than that. The saving grace is not that there is a lot of memory bandwidth to go around; there isn't; the saving grace is that a lot of processing either doesn't use too much data (making caches effective) or is complex enough to not be limited by memory bandwidth. Rarely using all cores and threads at the same time helps a lot as well.
  - dragontamer 7y ago
    > Actually it's super easy, barely an inconvenience Wow wow wow. Your references are TIGHT. ---------- The main RAM / CPU issue is latency: it takes over 100-clock cycles to communicate with DDR4, which a CPU could process ~400 instructions if they were lined up just right (modern CPUs are super-scalar, executing multiple instructions per clock tick). Only once you have sizable caches + data locality will you solve the latency problem. Without locality, I'm not sure if the latency problem can be solved at all. Fortunately, many problems have an element of locality that can be taken advantage of.
    - fulafel 7y ago
      Memory controllers do prefetching for streaming memory accesses. Yes a lot of workloads are more latency sensitive, but it's not like sequential streaming memory accesses are hard given the prefetching. (Anyone have pointers to current EPYC vs Xeon STREAM results?)
  - 7y ago
mschuetz 7y ago
It definitely gets harder to code in a way that is not memory bandwidth bound.

cma 7y ago

It has a separate dedicated IO die, allowing memory access/latency to all cores. I think threadripper does have less bandwidth than the 64 core Epyc part though.

amarshall 7y ago
Indeed as TR is 4-channel and Epyc is 8-channel. If one needs ECC, TR is a bit more limited still due to unbuffered ECC memory having lower max speeds presently.
- cma 7y ago
  Can't edit now but I meant to write "allowing [uniform] memory access/latency to all cores"

greglindahl 7y ago

CPUs keep on getting more memory channels per socket. That trend was already visible 15 years ago, when AMD moved the northbridge onto the CPU. Scaling it depends on scaling pin counts, which are now amazing.

wtallis 7y ago
To put some numbers to this: In 2005, Opterons used a 940-pin socket and had dual-channel DDR, for 6.4GB/s total bandwidth to feed two cores. In 2009, Xeons the LGA1366 socket to provide triple-channel DDR3, 32GB/s for four (later six) cores. Now, we're up to 4094 pin sockets providing 8 channels of DDR4 for a total of 204.8GB/s of memory bandwidth for 64 cores, which is exactly the same per-core bandwidth we had in 2005. (And for comparison, a $200 graphics card will also have around 200GB/s of memory bandwidth.)
- vbezhenar 7y ago
  Unrelated question, what about RAM latency? Did it significantly improve since 2005?
  - dwrodri 7y ago
    So at the consumer level, DDR2[popular standard c. 2005] could be clocked up to 266 MHz whereas DDR4[popular now] clocks in at 1600MHz. So a little more than a 6x speedup in transaction times. Unlike processors, whose features make it quite difficult to compare against each other, DRAM has stayed relatively straightforward: its still just a bunch of densely packed flip-flops connected to a memory controller that offers bank-strided column/row access. As you can imagine, processor performance has grown a lot more than 6x over the years. The amount of cores in a package alone has increased more than 6x! So how do we get around this? 1) Larger caches: the most efficient optimization to any slow task is to simply not perform said task. Having a larger L3 cache means sending requesting fewer pages from DRAM. 2) “Hiding” latency with bandwidth: if I have to incur a (relatively) large time penalty tp go to DRAM, I better make sure to make it worth my time. Instead of always getting data in 4KiB pages, often times modern processors support “superpages” all the way up to the gigabytes. If you’re going to go drive out to the store, it makes sense to buy groceries for the week instead of just the candy bar you’re craving right now, right? That way when you actually have to cook the work has already been done. I am probably missing some nuance here, especially when it comes to memory controllers and putting caches on DRAM, but this is the gist.
    - vbezhenar 7y ago
      But frequency of RAM and latency is not the same. DDR2-800 can have something like 4-4-4-12 timings, while DDR4-2666 can have something like 15-17-17-35 timings. So their result latency will be around the same! And those numbers I just googled, I did not made them up.
    - Retric 7y ago
      DDR2 vs DDR4 has effectively zero impact on latency. Due to the physical distances between RAM modules and the CPU, that’s been stuck within a narrow range for decades. https://en.wikipedia.org/wiki/CAS_latency https://en.wikipedia.org/wiki/CAS_latency The specific implication details make a difference, but nothing is going to provide a 4x improvement in latency so other considerations take precedence. The real change is ever growing cache sizes allowing CPU’s to minimize random RAM reads.

codeflo 7y ago

I’m interested in this as well. As opposed to CPUs from 15 years ago, the 3990X has 256 MB L3 cache and around 100 GB/s of memory bandwidth, which should help a bit. What kinds of workloads can we expect to be bandwidth-botllenecked still?

BubRoss 7y ago
If a program is structured in a particular way to run through large spans of contiguous memory but only do trivial operations, then memory bandwidth can be maxed out by the prefetcher and the cache wouldn't be used. Things like adding two huge images with SIMD and writing out a new image could be like this. It would be much better of course if the image is tiled and more operations are done per tile, which would use the cache. Good use of the prefetcher but poor use of the cache could still run through memory bandwdith.

imtringued 7y ago

The rule of thumb is 25GB/s per channel. At 8 channels you have 200GB/s. Unless you are working with SIMD it is unlikely that you actually hit these numbers.

blattimwind 7y ago
Actually it's super easy, barely an inconvenience, since the raw memory bandwidth you get per core is around 1 byte per cycle and half that per thread. A 3900X with 12 cores / 24 threads getting a total memory bandwidth of 57.6 GB/s (using DDR4-3600) has just 0.6 bytes / cycle and thread. Meanwhile the CPU can actually achieve ALU throughputs that are 20 times greater than that. Even encryption algorithms are much faster than that. The saving grace is not that there is a lot of memory bandwidth to go around; there isn't; the saving grace is that a lot of processing either doesn't use too much data (making caches effective) or is complex enough to not be limited by memory bandwidth. Rarely using all cores and threads at the same time helps a lot as well.
- dragontamer 7y ago
  > Actually it's super easy, barely an inconvenience Wow wow wow. Your references are TIGHT. ---------- The main RAM / CPU issue is latency: it takes over 100-clock cycles to communicate with DDR4, which a CPU could process ~400 instructions if they were lined up just right (modern CPUs are super-scalar, executing multiple instructions per clock tick). Only once you have sizable caches + data locality will you solve the latency problem. Without locality, I'm not sure if the latency problem can be solved at all. Fortunately, many problems have an element of locality that can be taken advantage of.
  - fulafel 7y ago
    Memory controllers do prefetching for streaming memory accesses. Yes a lot of workloads are more latency sensitive, but it's not like sequential streaming memory accesses are hard given the prefetching. (Anyone have pointers to current EPYC vs Xeon STREAM results?)
- 7y ago

mschuetz 7y ago

It definitely gets harder to code in a way that is not memory bandwidth bound.