4 ms·

I try to refrain from purely negative commentary on articles, but yuck! How can one hope to say anything useful about "memory performance" without once using t

by nkurz 11y ago

I try to refrain from purely negative commentary on articles, but yuck! How can one hope to say anything useful about "memory performance" without once using the terms "latency" or "bandwidth"? Memory performance is getting higher in the same way that processor performance is going up: http://www.corsair.com/en-us/blog/2015/september/ddr3_vs_ddr4_generational http://www.corsair.com/en-us/blog/2015/september/ddr3_vs_ddr... Latency is to frequency as bandwidth is to parallelism. Single core CPU frequencies are relatively stable, but parallelism offers the opportunity to get more done each cycle. Latency for random access is holding quite steady, but caches are getting bigger and faster, and bandwidth is going up. The key is figuring out how to write software that takes advantage of spatial locality and available bandwidth rather than getting choked by the latency. This is hard in the same way that taking advantage of multiple cores is hard: it requires a different approach, but is not a fundamental limitation. Lots of memory access isn't the problem. Long latency isn't the problem. The problem is designing your program so that it generates lots of long latency memory accesses and then grinds to a halt in the presence of this latency. I think the summary from this recent paper is spot on: Our conclusion is contrary to our expectations and to previous findings and goes against conventional wisdom, which states that accesses to RAM are slow, and should be minimized. A more accurate statement, that accounts for our findings, is that accesses to RAM have high latency and this latency needs to be mitigated. http://arxiv.org/pdf/1509.05053v1.pdf http://arxiv.org/pdf/1509.05053v1.pdf

vardump 11y ago
> I try to refrain from purely negative commentary on articles, but yuck! How can one hope to say anything useful about "memory performance" without once using the terms "latency" or "bandwidth"? Yeah, it is pretty clear the author didn't really understand much of the topic. Even if you have lightning fast memory, there's still the issue of connecting it. Long traces and multiple levels of multiplexing are going to add latency no matter how amazing memory tech is being employed. Before long, to increase bandwidth, memory needs to be integrated in the same package as the CPU if not on the same die. At least large eDRAM-style 'caches'. You can't economically run 1024 traces just for memory on the mainboard PCB! > The key is figuring out how to write software that takes advantage of spatial locality and available bandwidth rather than getting choked by the latency. This is hard in the same way that taking advantage of multiple cores is hard: it requires a different approach, but is not a fundamental limitation. Indeed, of course this requires skilled labor for now, until compilers catch up one day. And unfortunately code that truly requires random access is just not going to perform well on modern hardware. Up until early nineties, memory was faster than processors. Random access was just fine. Not so anymore. It's also easy to get bandwidth limited with SSE and AVX. Although line fill etc. buffers seem to often bottleneck first per CPU core. For scalar code, being bandwidth limited is not going to happen. The issue nowadays is that machines are so unique snowflakes performance and configuration wise. It's not hard to max out a single configuration, but it is hard to write something that performs decently across different system configurations. There are like 30 instruction set extensions for x86, variations in reorder buffer depth, variations in cache latency, size and associativity. And of course memory bus configurations, number and interleaving of memory channels, NUMA, DRAM memory page size (1, 2, 4 kB), etc. For example unlike 8-way associative L2 cache on Sandy Bridge, Haswell, etc., on Skylake it is now 4-way instead. Code that is optimised for 8-way L2 cache might be pathologically invalidating L2 cache lines on Skylake. Those aspects matter, because high performance is often a balancing act between available features, bandwidth and CPU power.
- Filligree 11y ago
  And while you're at it, it's hard to figure out what's going on in your system. I've spent some time looking, and have not yet found anything resembling a bandwidth monitor for main memory.
  - nkurz 11y ago
    It's possible, but depending on your OS and processor it might be quite difficult. There are "Uncore" performance counters for this, but the means of access has been changing from generation to generation and the software sometimes lags behind. For Linux and Haswell EP, I've had luck with likwid: https://github.com/RRZE-HPC/likwid/wiki/Haswell-EP#memory-controller-fixed-purpose-counters https://github.com/RRZE-HPC/likwid/wiki/Haswell-EP#memory-co... You also might have luck with Andi Kleen's pmu-tools: https://github.com/andikleen/pmu-tools/blob/master/ucevent/README.md https://github.com/andikleen/pmu-tools/blob/master/ucevent/R... I don't know for sure if Intel's VTune supports these counters, but I'd presume it would: https://software.intel.com/en-us/intel-vtune-amplifier-xe https://software.intel.com/en-us/intel-vtune-amplifier-xe
    - Filligree 11y ago
      Doesn't seem like any of them work with my E3 Xeon.
h1290799 11y ago
There is nothing hard about this. You either have a "problem" that can take advantage of multiple cores or sequential reading from memory or you don't. The only way to avoid this is to change the "problem"
- MichaelGG 11y ago
  Many times you'll have code that doesn't take advantage of multiple chores and sequential memory, but could. Redesigning it is what's hard.
ghshephard 11y ago
Just as a side note - after working at an RF Networking company for 10+ years, the phrase "bandwidth" (which in RF, is literally the "width of the band", e.g. 200 kHz ) is endlessly confusing, particularly when you hear engineers use phrases like, "we've got more bandwidth resulting in greater range but lower data rate with this signaling scheme." The RF PHY/MAC engineers use data rate, and bandwidth 100% consistently, and everyone else I've ever met, uses "bandwidth" as a synonym for "data rate" - I'm surprised there isn't more confusion between the two groups of people. At least all groups measures data rate and "bandwidth" with megabits/sec == 1x10e6 bits/second. Thankfully that confusion never entered the data comms world.

by nkurz 11y ago

vardump 11y ago

> I try to refrain from purely negative commentary on articles, but yuck! How can one hope to say anything useful about "memory performance" without once using the terms "latency" or "bandwidth"? Yeah, it is pretty clear the author didn't really understand much of the topic. Even if you have lightning fast memory, there's still the issue of connecting it. Long traces and multiple levels of multiplexing are going to add latency no matter how amazing memory tech is being employed. Before long, to increase bandwidth, memory needs to be integrated in the same package as the CPU if not on the same die. At least large eDRAM-style 'caches'. You can't economically run 1024 traces just for memory on the mainboard PCB! > The key is figuring out how to write software that takes advantage of spatial locality and available bandwidth rather than getting choked by the latency. This is hard in the same way that taking advantage of multiple cores is hard: it requires a different approach, but is not a fundamental limitation. Indeed, of course this requires skilled labor for now, until compilers catch up one day. And unfortunately code that truly requires random access is just not going to perform well on modern hardware. Up until early nineties, memory was faster than processors. Random access was just fine. Not so anymore. It's also easy to get bandwidth limited with SSE and AVX. Although line fill etc. buffers seem to often bottleneck first per CPU core. For scalar code, being bandwidth limited is not going to happen. The issue nowadays is that machines are so unique snowflakes performance and configuration wise. It's not hard to max out a single configuration, but it is hard to write something that performs decently across different system configurations. There are like 30 instruction set extensions for x86, variations in reorder buffer depth, variations in cache latency, size and associativity. And of course memory bus configurations, number and interleaving of memory channels, NUMA, DRAM memory page size (1, 2, 4 kB), etc. For example unlike 8-way associative L2 cache on Sandy Bridge, Haswell, etc., on Skylake it is now 4-way instead. Code that is optimised for 8-way L2 cache might be pathologically invalidating L2 cache lines on Skylake. Those aspects matter, because high performance is often a balancing act between available features, bandwidth and CPU power.

Filligree 11y ago
And while you're at it, it's hard to figure out what's going on in your system. I've spent some time looking, and have not yet found anything resembling a bandwidth monitor for main memory.
- nkurz 11y ago
  It's possible, but depending on your OS and processor it might be quite difficult. There are "Uncore" performance counters for this, but the means of access has been changing from generation to generation and the software sometimes lags behind. For Linux and Haswell EP, I've had luck with likwid: https://github.com/RRZE-HPC/likwid/wiki/Haswell-EP#memory-controller-fixed-purpose-counters https://github.com/RRZE-HPC/likwid/wiki/Haswell-EP#memory-co... You also might have luck with Andi Kleen's pmu-tools: https://github.com/andikleen/pmu-tools/blob/master/ucevent/README.md https://github.com/andikleen/pmu-tools/blob/master/ucevent/R... I don't know for sure if Intel's VTune supports these counters, but I'd presume it would: https://software.intel.com/en-us/intel-vtune-amplifier-xe https://software.intel.com/en-us/intel-vtune-amplifier-xe
  - Filligree 11y ago
    Doesn't seem like any of them work with my E3 Xeon.

h1290799 11y ago

There is nothing hard about this. You either have a "problem" that can take advantage of multiple cores or sequential reading from memory or you don't. The only way to avoid this is to change the "problem"

MichaelGG 11y ago
Many times you'll have code that doesn't take advantage of multiple chores and sequential memory, but could. Redesigning it is what's hard.

ghshephard 11y ago

Just as a side note - after working at an RF Networking company for 10+ years, the phrase "bandwidth" (which in RF, is literally the "width of the band", e.g. 200 kHz ) is endlessly confusing, particularly when you hear engineers use phrases like, "we've got more bandwidth resulting in greater range but lower data rate with this signaling scheme." The RF PHY/MAC engineers use data rate, and bandwidth 100% consistently, and everyone else I've ever met, uses "bandwidth" as a synonym for "data rate" - I'm surprised there isn't more confusion between the two groups of people. At least all groups measures data rate and "bandwidth" with megabits/sec == 1x10e6 bits/second. Thankfully that confusion never entered the data comms world.