5 ms·

Memory bandwidth is becoming the ultimate spec. In related news, Apple reduced memory bandwidth in their M3 chips by 25% compared to M1/M2: https://www.macrumor

by krasin 3y ago

Memory bandwidth is becoming the ultimate spec. In related news, Apple reduced memory bandwidth in their M3 chips by 25% compared to M1/M2: https://www.macrumors.com/2023/10/31/apple-m3-pro-less-memory-bandwidth/ https://www.macrumors.com/2023/10/31/apple-m3-pro-less-memor...

fswd 3y ago
I'm so glad we're seeing memory specs next to the core count, awesome.
Aurornis 3y ago
> Memory bandwidth is becoming the ultimate spec. Only in applications that are constrained by memory bandwidth. Many applications are not constrained by the memory bandwidth on modern architectures with significant cache. If you have an application constrained by memory bandwidth, carefully selecting the server would make sense. > In related news, Apple reduced memory bandwidth in their M3 chips by 25% compared to M1/M2: Not exactly. M3 Max has the same 400GB/sec memory bandwidth as the top end M1 and M2 chips. It’s only certain lower tiers that have less memory bandwidth than their equivalent tiers in previous gens. But it probably doesn’t matter for most applications.
- admax88qqq 3y ago
  > Only in applications that are constrained by memory bandwidth. Sure but I think OPs point is that memory bandwidth is the bottleneck in more applications that you think. Cache only helps if you're accessing the same data over and over. My gut instinct is the amount of applications that do a lot of processing but only on a small amount of data are in the minority. The opposite seems a lot more common.
  - jltsiren 3y ago
    I would assume that applications that process data faster than a few gigabytes per CPU-second are rare. It's more common that things are constrained by computation or memory latency.
    - Xorlev 3y ago
      You'd think so, but for datacenter workloads it's absolutely common, especially if you're just scheduling a bunch of containers together. Computation also doesn't happen in a vacuum, unless you're doing some fairly trivial processing you're likely loading quite a bit of memory, perhaps many multiples of what your business logic is actually doing. It's also not as easy as GB/s/core, since cores aren't entirely uniform, and data access may be across core complexes.
      - jltsiren 3y ago
        I'm not sure what you mean by datacenter workloads. The work I do could be called data science and data engineering. Outside some fairly trivial (or highly optimized) sequential processing, the CPU just isn't fast enough to saturate memory bandwidth. For anything more complex, the data you want to load is either in cache (and bandwidth doesn't matter) or it isn't (and you probably care more about latency).
        terlisimo 3y ago
        I had these two dual-18-core xeon web servers with seemingly identical hardware and software setup but one was doing 1100 req/s and the other 500-600. After some digging, I've realized that one had 8x8GB ram modules and the slower one had 2x32GB. I did some benchmarking then and found that it really depends on the workload. The www app was 50% slower. Memcache 400% slower. Blender 5% slower. File compression 20%. Most single-threaded tasks no difference. The takeaway was that workloads want some bandwidth per core, and shoving more cores into servers doesn't increase performance once you hit memory bandwidth limits.
        mirsadm 3y ago
        This seems very unlikely. The CPU is almost always bottlenecked by memory.
        jltsiren 3y ago
        It's usually bottlenecked by memory latency, not bandwidth. People talk about bandwidth, because it's a simple number that keeps growing over time. Latency stays at ~100 ns, because DRAM is not getting any faster. Bandwidth can become a real constraint if your single-threaded code is processing more than a couple of gigabytes per second. But it usually takes a lot of micro-optimization to do anything meaningful at such speeds.
  - eightnoteight 3y ago
    exactly, a typical REST server spends a lot of compute time in json encoding and decoding. which could be sped up significantly if there is better memory bandwidth
    - glhaynes 3y ago
      My first guess would've been that JSON encode/decode wouldn't be memory bandwith-bound because it involves so many operations on single bytes at a time, most of which would be in the cache after the page has been brought in initially. Of course maybe much of that is done with SIMD these days? Anyway, I have near-zero experience in this area so I'm mostly just posting this hoping for someone to explain why I'm wrong.
      - densh 3y ago
        See https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson
        gpderetta 3y ago
        I don't see how the link clarifies things. simdjson is fast enough that JSON decoding can approach single-core bandwidth capacity, but it doesn't follow that a JSON-heavy load is bandwitdh constrained. In fact the existence of simdjson points to json decoding being compute bound.
        densh 3y ago
        It’s a bit more nuanced than that. Scalar code that does lots of branching and loads one value at a time (e.g. single character) often can’t hit memory at peak bandwidth rate. On the other hand well vectorized code that loads vectors of multiple values at a time can properly saturate the available hardware resources.
        glhaynes 3y ago
        The earlier post was referring to "a typical REST server" in a context of the claim that memory bandwidth "is becoming the ultimate spec". simdjson (impressive as it may be) existing as an interesting project seems likely to imply that most JSON parsing isn't done with such performant methods on "the typical REST server". EDIT: Seeing that it's used in many projects including Node.js shifts me back to thinking more highly of the claim that memory bandwidth is becoming the ultimate spec!
  - TerrifiedMouse 3y ago
    > My gut instinct is the amount of applications that do a lot of processing but only on a small amount of data are in the minority. The opposite seems a lot more common. I doubt that. The easily way to find out is to disable the L3 cache on your CPU. If your theory is right, the performance drop should be minimal on most applications. Note: Even with the L3 cache disabled there is still the L1 and L2 caches but those are pretty small.
    - vilunov 3y ago
      I suspect that L3 impacts performance a lot. I dabbled in running a Windows VM with a GPU passthrough, and I had to spend a lot of time to properly pin virtual cores to physical ones so that CPU caches are utilized properly, as I suspected that it was the reason for decreased performance. I managed to get only L1 and L2 to be utilized properly and it somewhat helped, but L3 was still underutilized due to double-scheduling in both Win VM and Linux hypervisor. Even the basic OS shell suffered from lags, not to mention games.
ChuckMcM 3y ago
Narrators voice: It was always the ultimate spec, it is now becoming more obvious to more people. One of the interesting things for me is how memory "operations" (which is to say transactions of the memory controller) can completely obliterate "bandwidth." In the past (not sure how true this is on current microarchitectures) the controller between cache and DRAM worked by opening a "page" of dynamic memory. That was an artifact of how DRAMs use both a column address select and a row address select and then multiplex the address bits. So to "read" memory you needed to select the column, then select the row, and then read the memory. The good news was that if you needed the next word of memory in order, you could just read again. And again. Periodically if you were reading the chip would have to ask you to wait while it refreshed its contents. Anyway, this memory operation of opening a new page was a lot slower than reading the next word in memory. So if you're requests were bouncing all around memory your effective bandwidth was limited by how fast your memory controller could open new pages. That could be one tenth the nominal serial access bandwidth. Controllers had multiple "page" registers so they could hold the state of two (or more) different DIMMS and try to interleave their access across DIMMs to hide the latency aspect of page mechanics. Generally though, when you get to the point where your measuring memops and trying to layout your physical memory to minimize them you're in a different realm of system optimization.
jjtheblunt 3y ago
I think Seymour Cray significantly prioritized this in his designs. The emphasis on processor bandwidth on memory even dates back to the CDC Star. https://en.wikipedia.org/wiki/Cray-1 https://en.wikipedia.org/wiki/Cray-1
charleshn 3y ago
It's a shame more people aren't familiar with arithmetic intensity and the roofline model [1], which give a simple mental model for performance profiles on both CPU and GPU. I always find this graph [2] covering decades of hardware evolution fascinating. [1] https://en.m.wikipedia.org/wiki/Roofline_model https://en.m.wikipedia.org/wiki/Roofline_model [2] http://www.nextplatform.com/wp-content/uploads/2022/12/dongarra-turing-memory-vs-compute.jpg http://www.nextplatform.com/wp-content/uploads/2022/12/donga...
fulafel 3y ago
"Becoming the ultimate spec" is one perspective, another is "the drive toward increasingly bandwidth-starved cores starts to hit limits in some applications". Bandwidth per core has been decreasing due to increasing core counts and slowly improving per socket mem BW, and cores have gotten faster at compute. Anyone know some example bytes per flop machine balance numbers for current and eg 20 year old systems? For older ones one source is https://www.cs.virginia.edu/stream/peecee/Balance.html https://www.cs.virginia.edu/stream/peecee/Balance.html - the "machine balance" for 2003 boxes seems to be between 10 and 17 there. Sadly the "MW/s" unit for machine words is not self-explanatory, what is the word size used.
GeekyBear 3y ago
Given that the "reduced" bandwidth is still more than double that of an Intel or AMD HEDT chip, you're still doing much better than you would with x86. Intel and AMD both tend to choke bandwidth (memory and PCI lanes) on anything below their server chips.
seuraughty 3y ago
The high memory bandwidth is impressive, but you’d actually have to contrive some set of artificial circumstances to saturate it. In almost all use cases the user won’t tell the difference. https://www.anandtech.com/print/17024/apple-m1-max-performance-review https://www.anandtech.com/print/17024/apple-m1-max-performan...

by krasin 3y ago

fswd 3y ago

I'm so glad we're seeing memory specs next to the core count, awesome.

Aurornis 3y ago

> Memory bandwidth is becoming the ultimate spec. Only in applications that are constrained by memory bandwidth. Many applications are not constrained by the memory bandwidth on modern architectures with significant cache. If you have an application constrained by memory bandwidth, carefully selecting the server would make sense. > In related news, Apple reduced memory bandwidth in their M3 chips by 25% compared to M1/M2: Not exactly. M3 Max has the same 400GB/sec memory bandwidth as the top end M1 and M2 chips. It’s only certain lower tiers that have less memory bandwidth than their equivalent tiers in previous gens. But it probably doesn’t matter for most applications.

admax88qqq 3y ago
> Only in applications that are constrained by memory bandwidth. Sure but I think OPs point is that memory bandwidth is the bottleneck in more applications that you think. Cache only helps if you're accessing the same data over and over. My gut instinct is the amount of applications that do a lot of processing but only on a small amount of data are in the minority. The opposite seems a lot more common.
- jltsiren 3y ago
  I would assume that applications that process data faster than a few gigabytes per CPU-second are rare. It's more common that things are constrained by computation or memory latency.
  - Xorlev 3y ago
    You'd think so, but for datacenter workloads it's absolutely common, especially if you're just scheduling a bunch of containers together. Computation also doesn't happen in a vacuum, unless you're doing some fairly trivial processing you're likely loading quite a bit of memory, perhaps many multiples of what your business logic is actually doing. It's also not as easy as GB/s/core, since cores aren't entirely uniform, and data access may be across core complexes.
    - jltsiren 3y ago
      I'm not sure what you mean by datacenter workloads. The work I do could be called data science and data engineering. Outside some fairly trivial (or highly optimized) sequential processing, the CPU just isn't fast enough to saturate memory bandwidth. For anything more complex, the data you want to load is either in cache (and bandwidth doesn't matter) or it isn't (and you probably care more about latency).
      - terlisimo 3y ago
        I had these two dual-18-core xeon web servers with seemingly identical hardware and software setup but one was doing 1100 req/s and the other 500-600. After some digging, I've realized that one had 8x8GB ram modules and the slower one had 2x32GB. I did some benchmarking then and found that it really depends on the workload. The www app was 50% slower. Memcache 400% slower. Blender 5% slower. File compression 20%. Most single-threaded tasks no difference. The takeaway was that workloads want some bandwidth per core, and shoving more cores into servers doesn't increase performance once you hit memory bandwidth limits.
      - mirsadm 3y ago
        This seems very unlikely. The CPU is almost always bottlenecked by memory.
        jltsiren 3y ago
        It's usually bottlenecked by memory latency, not bandwidth. People talk about bandwidth, because it's a simple number that keeps growing over time. Latency stays at ~100 ns, because DRAM is not getting any faster. Bandwidth can become a real constraint if your single-threaded code is processing more than a couple of gigabytes per second. But it usually takes a lot of micro-optimization to do anything meaningful at such speeds.
- eightnoteight 3y ago
  exactly, a typical REST server spends a lot of compute time in json encoding and decoding. which could be sped up significantly if there is better memory bandwidth
  - glhaynes 3y ago
    My first guess would've been that JSON encode/decode wouldn't be memory bandwith-bound because it involves so many operations on single bytes at a time, most of which would be in the cache after the page has been brought in initially. Of course maybe much of that is done with SIMD these days? Anyway, I have near-zero experience in this area so I'm mostly just posting this hoping for someone to explain why I'm wrong.
    - densh 3y ago
      See https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson
      - gpderetta 3y ago
        I don't see how the link clarifies things. simdjson is fast enough that JSON decoding can approach single-core bandwidth capacity, but it doesn't follow that a JSON-heavy load is bandwitdh constrained. In fact the existence of simdjson points to json decoding being compute bound.
        densh 3y ago
        It’s a bit more nuanced than that. Scalar code that does lots of branching and loads one value at a time (e.g. single character) often can’t hit memory at peak bandwidth rate. On the other hand well vectorized code that loads vectors of multiple values at a time can properly saturate the available hardware resources.
        glhaynes 3y ago
        The earlier post was referring to "a typical REST server" in a context of the claim that memory bandwidth "is becoming the ultimate spec". simdjson (impressive as it may be) existing as an interesting project seems likely to imply that most JSON parsing isn't done with such performant methods on "the typical REST server". EDIT: Seeing that it's used in many projects including Node.js shifts me back to thinking more highly of the claim that memory bandwidth is becoming the ultimate spec!
- TerrifiedMouse 3y ago
  > My gut instinct is the amount of applications that do a lot of processing but only on a small amount of data are in the minority. The opposite seems a lot more common. I doubt that. The easily way to find out is to disable the L3 cache on your CPU. If your theory is right, the performance drop should be minimal on most applications. Note: Even with the L3 cache disabled there is still the L1 and L2 caches but those are pretty small.
  - vilunov 3y ago
    I suspect that L3 impacts performance a lot. I dabbled in running a Windows VM with a GPU passthrough, and I had to spend a lot of time to properly pin virtual cores to physical ones so that CPU caches are utilized properly, as I suspected that it was the reason for decreased performance. I managed to get only L1 and L2 to be utilized properly and it somewhat helped, but L3 was still underutilized due to double-scheduling in both Win VM and Linux hypervisor. Even the basic OS shell suffered from lags, not to mention games.

ChuckMcM 3y ago

Narrators voice: It was always the ultimate spec, it is now becoming more obvious to more people. One of the interesting things for me is how memory "operations" (which is to say transactions of the memory controller) can completely obliterate "bandwidth." In the past (not sure how true this is on current microarchitectures) the controller between cache and DRAM worked by opening a "page" of dynamic memory. That was an artifact of how DRAMs use both a column address select and a row address select and then multiplex the address bits. So to "read" memory you needed to select the column, then select the row, and then read the memory. The good news was that if you needed the next word of memory in order, you could just read again. And again. Periodically if you were reading the chip would have to ask you to wait while it refreshed its contents. Anyway, this memory operation of opening a new page was a lot slower than reading the next word in memory. So if you're requests were bouncing all around memory your effective bandwidth was limited by how fast your memory controller could open new pages. That could be one tenth the nominal serial access bandwidth. Controllers had multiple "page" registers so they could hold the state of two (or more) different DIMMS and try to interleave their access across DIMMs to hide the latency aspect of page mechanics. Generally though, when you get to the point where your measuring memops and trying to layout your physical memory to minimize them you're in a different realm of system optimization.

jjtheblunt 3y ago

I think Seymour Cray significantly prioritized this in his designs. The emphasis on processor bandwidth on memory even dates back to the CDC Star. https://en.wikipedia.org/wiki/Cray-1 https://en.wikipedia.org/wiki/Cray-1

charleshn 3y ago

It's a shame more people aren't familiar with arithmetic intensity and the roofline model [1], which give a simple mental model for performance profiles on both CPU and GPU. I always find this graph [2] covering decades of hardware evolution fascinating. [1] https://en.m.wikipedia.org/wiki/Roofline_model https://en.m.wikipedia.org/wiki/Roofline_model [2] http://www.nextplatform.com/wp-content/uploads/2022/12/dongarra-turing-memory-vs-compute.jpg http://www.nextplatform.com/wp-content/uploads/2022/12/donga...

fulafel 3y ago

"Becoming the ultimate spec" is one perspective, another is "the drive toward increasingly bandwidth-starved cores starts to hit limits in some applications". Bandwidth per core has been decreasing due to increasing core counts and slowly improving per socket mem BW, and cores have gotten faster at compute. Anyone know some example bytes per flop machine balance numbers for current and eg 20 year old systems? For older ones one source is https://www.cs.virginia.edu/stream/peecee/Balance.html https://www.cs.virginia.edu/stream/peecee/Balance.html - the "machine balance" for 2003 boxes seems to be between 10 and 17 there. Sadly the "MW/s" unit for machine words is not self-explanatory, what is the word size used.

GeekyBear 3y ago

Given that the "reduced" bandwidth is still more than double that of an Intel or AMD HEDT chip, you're still doing much better than you would with x86. Intel and AMD both tend to choke bandwidth (memory and PCI lanes) on anything below their server chips.

seuraughty 3y ago

The high memory bandwidth is impressive, but you’d actually have to contrive some set of artificial circumstances to saturate it. In almost all use cases the user won’t tell the difference. https://www.anandtech.com/print/17024/apple-m1-max-performance-review https://www.anandtech.com/print/17024/apple-m1-max-performan...