6 ms·

> most workloads could actually run more efficiently if they had more memory bandwidth and lower latency access to memory Turns out memory access speed is more

> most workloads could actually run more efficiently if they had more memory bandwidth and lower latency access to memory Turns out memory access speed is more or less the entire game for everything except scientific computing or insanely optimized code. In the real world, CPU frequency seems to matter much less than DRAM timings, for example, in everything but extremely well engineered games. It'll be interesting to learn (if we ever do) how much of the "real-world" 25% performance gain is solely due to DDR5. I remember getting my AMD K8 Opteron around 2003 or 2004 with the first on-die memory controller. Absolutely demolished Intel chips at the time in non-synthetic benchmarks.

shoo 5y ago
> everything except scientific computing or insanely optimized code for insanely unoptimized code, such as accidentally ending up writing something compute intensive in pure python, its very plausible for it to be compute constrained -- but less because of the hardware and more because 99 %or 99.9% of the operations you're asking the cpu to perform are effectively waste.
- erulabs 5y ago
  Certainly, but many "wasteful" operations are reading/writing main memory unnecessarily - and since a single trip out to memory and back can take many many CPU cycles, typically optimizing memory access time does more for "bad code" than optimizing the number of CPU cycles per second. But obviously, you're right too - faster is faster after all :)
- throwaway894345 5y ago
  I'm skeptical of this. I would expect many of those wasted instructions to be expensive memory instructions. I'm pretty sure regular Python code allocates a lot and the interpreter spends lots of time chasing after memory references.
hinkley 5y ago
In the 90’s there were people trying to solve this problem by putting a small CPU on chip with the memory and running some operations there. I routinely wonder why memory hasn’t gotten smarter over time.
- y4mi 5y ago
  DDR5 has basically minor cpus on the RAM for improved performance. That's the main reason why they're so expensive and sometimes need active cooling
  - Terry_Roll 5y ago
    So like a (micro)SD card that has some arm cpu for wear levelling and other performance enhancements. It sounds interesting. https://www.bunniestudios.com/blog/?p=3554 https://www.bunniestudios.com/blog/?p=3554
  - GordonS 5y ago
    Interesting - do you know what kind of performance gains DDR5 will provide for real-life workloads? Having active cooling, making the system noisier and potentially hotter, seems like a pretty big down side.
    - vlangber 5y ago
      Anandtech compared DDR4 and DDR5 performance in their Alder Lake review: https://www.anandtech.com/show/17047/the-intel-12th-gen-core-i912900k-review-hybrid-performance-brings-hybrid-complexity/6 https://www.anandtech.com/show/17047/the-intel-12th-gen-core...
- eropple 5y ago
  Distributing work to leverage faster memory locality is hard. It's not quite what you're talking about, but consider the Cell processor used in the PS3 - the compute capability in HTPC space was supposed to be prodigious, but even having faster (streaming) access to RAM had the tradeoff of dealing with code dispatch. (It's not a perfect example, the SPE model was also just kind of a pain, but you have to think about how to get your code local to the memory you want, how to keep allocation nonrivalrous, etc. - it's a lot!)
klelatti 5y ago
The team that designed the original Arm CPU in 1985 came to the conclusion that bandwidth was the most important factor influencing performance - they even approached Intel for a 286 with more memory bandwidth!
baybal2 5y ago
Memory bandwidth is not a problem. Even puny 1 channel memory desktops don't usually saturate it. It's memory latency.
- Proven 5y ago
  That's the same thing
- mrcode007 5y ago
  Certain instruction sequences prohibit you from fully utilizing memory bandwidth and decrease it by as much as 40-60% so this statement is not true. For example, not using store and load buffers in ways they were designed to be used will lead to subpar performance for no apparent reason
  - ahartmetz 5y ago
    Can you elaborate? I have a side project(1) where all profilers I've used give a very muddled picture, so I'm very interested in the question of what slows down code on a modern "big" CPU with wide dispatch, a few kB of decoded ops buffer and a lot of OoO hardware. (1) It encodes and decodes a protocol from a potentially untrusted source, so there is obviously a lot of waiting for previous results. That much is clear, however I expected profilers to show me some causal link between serial nature and slow execution, but they don't. I have tried perf, Valgrind-Callgrind and AMD μProf (because I have a Ryzen CPU on both of my main private computers). I'm not sure if the tools suck, my test cases suck, or I just don't know how to interpret the tools' results - assignment of cost to lines of code seeming highly unreliable is my main problem. Maybe the stupid things (most of optimization is about not doing stupid things, after that it gets properly hard) that these profilers are designed to catch aren't the stupid or unavoidable things my code is doing.
    - namibj 5y ago
      KDAB's hotspot is quite nice for analyzing perf recordings, and I suggest looking at stall cycle and "cycles with less than X uops dispatched" events to sample on. Yes, attributing to lines in code is hard for optimized compiler output, but it can (in the continuous release/`hotspot-git` AUR package) attribute to the disassembly of a function.
mda 5y ago
I find this claim hard to believe honestly,could you point to examples where performance is limited by Dram speed and not by cpu / caches? They must be applications with extremely bad design causing super low cache hits.
- staticassertion 5y ago
  > They must be applications with extremely bad design causing super low cache hits. So basically any program written in a language with pointer types exclusively.
  - throwaway894345 5y ago
    I thought I’d heard that Java VMs go to great lengths to maintain cache coherence? I’d be curious to hear from the Lisp folks because I always hear that Lisps can be surprisingly performant.
    - staticassertion 5y ago
      There are some optimizations in the JVM that will improve cache coherence. Bump allocation helps, inlining and escape analysis help, etc. In theory the GC can also rearrange memory to 'compact' it. I'm not aware of this optimization in practice.
      - throwaway894345 5y ago
        As far as I know, all mainstream Java GCs are compacting; however, I don’t have an idea about the degree this improves cache coherence.
  - mda 5y ago
    Even then, today's cpus have enormous caches, and not all parts of program is pointers. you cant make a crappy application much faster just because you have faster ram.
- erulabs 5y ago

shoo 5y ago

> everything except scientific computing or insanely optimized code for insanely unoptimized code, such as accidentally ending up writing something compute intensive in pure python, its very plausible for it to be compute constrained -- but less because of the hardware and more because 99 %or 99.9% of the operations you're asking the cpu to perform are effectively waste.

erulabs 5y ago
Certainly, but many "wasteful" operations are reading/writing main memory unnecessarily - and since a single trip out to memory and back can take many many CPU cycles, typically optimizing memory access time does more for "bad code" than optimizing the number of CPU cycles per second. But obviously, you're right too - faster is faster after all :)
throwaway894345 5y ago
I'm skeptical of this. I would expect many of those wasted instructions to be expensive memory instructions. I'm pretty sure regular Python code allocates a lot and the interpreter spends lots of time chasing after memory references.

hinkley 5y ago

In the 90’s there were people trying to solve this problem by putting a small CPU on chip with the memory and running some operations there. I routinely wonder why memory hasn’t gotten smarter over time.

y4mi 5y ago
DDR5 has basically minor cpus on the RAM for improved performance. That's the main reason why they're so expensive and sometimes need active cooling
- Terry_Roll 5y ago
  So like a (micro)SD card that has some arm cpu for wear levelling and other performance enhancements. It sounds interesting. https://www.bunniestudios.com/blog/?p=3554 https://www.bunniestudios.com/blog/?p=3554
- GordonS 5y ago
  Interesting - do you know what kind of performance gains DDR5 will provide for real-life workloads? Having active cooling, making the system noisier and potentially hotter, seems like a pretty big down side.
  - vlangber 5y ago
    Anandtech compared DDR4 and DDR5 performance in their Alder Lake review: https://www.anandtech.com/show/17047/the-intel-12th-gen-core-i912900k-review-hybrid-performance-brings-hybrid-complexity/6 https://www.anandtech.com/show/17047/the-intel-12th-gen-core...
eropple 5y ago
Distributing work to leverage faster memory locality is hard. It's not quite what you're talking about, but consider the Cell processor used in the PS3 - the compute capability in HTPC space was supposed to be prodigious, but even having faster (streaming) access to RAM had the tradeoff of dealing with code dispatch. (It's not a perfect example, the SPE model was also just kind of a pain, but you have to think about how to get your code local to the memory you want, how to keep allocation nonrivalrous, etc. - it's a lot!)

klelatti 5y ago

The team that designed the original Arm CPU in 1985 came to the conclusion that bandwidth was the most important factor influencing performance - they even approached Intel for a 286 with more memory bandwidth!

baybal2 5y ago

Memory bandwidth is not a problem. Even puny 1 channel memory desktops don't usually saturate it. It's memory latency.

Proven 5y ago
That's the same thing
mrcode007 5y ago
Certain instruction sequences prohibit you from fully utilizing memory bandwidth and decrease it by as much as 40-60% so this statement is not true. For example, not using store and load buffers in ways they were designed to be used will lead to subpar performance for no apparent reason
- ahartmetz 5y ago
  Can you elaborate? I have a side project(1) where all profilers I've used give a very muddled picture, so I'm very interested in the question of what slows down code on a modern "big" CPU with wide dispatch, a few kB of decoded ops buffer and a lot of OoO hardware. (1) It encodes and decodes a protocol from a potentially untrusted source, so there is obviously a lot of waiting for previous results. That much is clear, however I expected profilers to show me some causal link between serial nature and slow execution, but they don't. I have tried perf, Valgrind-Callgrind and AMD μProf (because I have a Ryzen CPU on both of my main private computers). I'm not sure if the tools suck, my test cases suck, or I just don't know how to interpret the tools' results - assignment of cost to lines of code seeming highly unreliable is my main problem. Maybe the stupid things (most of optimization is about not doing stupid things, after that it gets properly hard) that these profilers are designed to catch aren't the stupid or unavoidable things my code is doing.
  - namibj 5y ago
    KDAB's hotspot is quite nice for analyzing perf recordings, and I suggest looking at stall cycle and "cycles with less than X uops dispatched" events to sample on. Yes, attributing to lines in code is hard for optimized compiler output, but it can (in the continuous release/`hotspot-git` AUR package) attribute to the disassembly of a function.

mda 5y ago

I find this claim hard to believe honestly,could you point to examples where performance is limited by Dram speed and not by cpu / caches? They must be applications with extremely bad design causing super low cache hits.

staticassertion 5y ago
> They must be applications with extremely bad design causing super low cache hits. So basically any program written in a language with pointer types exclusively.
- throwaway894345 5y ago
  I thought I’d heard that Java VMs go to great lengths to maintain cache coherence? I’d be curious to hear from the Lisp folks because I always hear that Lisps can be surprisingly performant.
  - staticassertion 5y ago
    There are some optimizations in the JVM that will improve cache coherence. Bump allocation helps, inlining and escape analysis help, etc. In theory the GC can also rearrange memory to 'compact' it. I'm not aware of this optimization in practice.
    - throwaway894345 5y ago
      As far as I know, all mainstream Java GCs are compacting; however, I don’t have an idea about the degree this improves cache coherence.
- mda 5y ago
  Even then, today's cpus have enormous caches, and not all parts of program is pointers. you cant make a crappy application much faster just because you have faster ram.
erulabs 5y ago