7 ms·

In short, yes, higher capacity will involve higher latency. The thing here is balancing core speed vs “effective” memory latency (factoring in cache hits, miss

In short, yes, higher capacity will involve higher latency. The thing here is balancing core speed vs “effective” memory latency (factoring in cache hits, misses, etc). Cache management is a hard problem, and the equilibrium point is load dependent (i.e. depends on what type of program you use). AMD has been smart enough to understand that sometimes it’s better to just brute force your way in (with higher cache sizes) than being super clever about how you handle it.

hajile 5y ago
AMD's SenseMI is a neural network to predict data fetching. That's quite the opposite of brute force. They add massive caches as a side effect of having loads of cores enabled by their chipset designs.
- wallacoloo 5y ago
  Depends on your definition of “brute force”, I guess. Between explicitly deriving some algorithm from an understanding of causal relationships, v.s. training a neural net on billions of samples without encoding any broader understanding of the world, the latter approach is the one I would have called “brute force”.
- xondono 5y ago
  Using NN for cache management is not that new, and is a simpler approach in comparison to Intels as far as I know.
IgorPartola 5y ago
At 192MB, just give me explicit access to this cache as memory and I’ll decide what goes into it. No need to try to predict what my code will want, I can just tell you.
- derefr 5y ago
  Imagine coding for a 64-bit version of the 6502, where you have a 192MB "zero page" :)
- BBC-vs-neolibs 5y ago
  If you boot on actual hardware (no VM), if your application code + kernel (or unikernel) fits in under 192 megabytes, you are golden.
- VHRanger 5y ago
  Put everything in contiguous memory and load it, it'll be in cache
  - IgorPartola 5y ago
    That not a great solution. My working data set is likely larger than 192MB. But my application is much more likely to know what data is in the hot path than the CPU is to guess it. I might want to put my DB index into this cache, or I might want to put only a part of it. I might want to preload the data then work on it for a bit, then load the next chunk. I might want one core preparing the data in RAM while another core is getting ready to work on it in this cache. Essentially I want full access to it independent of the main RAM because I can do a lot more with that. Think of this as the difference between RAM and disk: is RAM only useful as a cache to what’s on disk?
    - vlovich123 5y ago
      We know both positions are correct in practice. Most automatic prediction/management algorithms perform better on 90% of the code that’s run on them. However, careful performance tuning over time by experts can outperform generic solutions, sometimes dramatically. I generally err on the side of “I’m not as smart as I think I am” in these kinds of discussions. It’s not that I can never do it. I’m sure if I studied it a bit I could. The reason is that 99% of the lines of code I’ve ever written don’t warrant that kind of attention. I’m sure there are uses like when you’re writing some core fundamental algorithm in the fields of compression, crypto algorithm, hash, video processing, etc etc etc. I don’t work in those spaces though so the benefit is much more marginal. It’s possible 192mb is enough to start needing some explicit memory management. It makes the coding model much more complex though. And in a greedy software system where your code isn’t the only one running, such complexity doesn’t necessarily net overall wins. It’s the reason we have drivers and OSes even though we started with each piece of content bundling explicit HW support (at much better perf generally)
bcrl 5y ago
Actually, this is where 3d stacking changes the game. With 1 additional vertical layer, you can get double the memory at practically the same latency, as the source of latency is related to the wire distance (and capacitance) in the 2d plane. Latency can be lowered with 3d stacking as well if you reduce the 2d area of the memory array. This is how the industry will keep scaling going when 2d feature sizes can no longer be shrunk.
- davuinci 5y ago
  What's the limitation on adding additional vertical layers though? One apparent limitation could be that by adding several vertical layers it would require the wire to be longer.
  - cogman10 5y ago
    2 limitations. 1. Heat doesn't go away with layers and gets harder to cool. 2. These are still layers. You want each layer to be as flat as possible. As the layers deform you lose the ability to cleanly add more layers. Current lithography is very much based on the notion that everything is flat.
  - klodolph 5y ago
    Heat dissipation, although the power consumption of SRAM is mostly affected by I/O rather than capacity.

hajile 5y ago

AMD's SenseMI is a neural network to predict data fetching. That's quite the opposite of brute force. They add massive caches as a side effect of having loads of cores enabled by their chipset designs.

wallacoloo 5y ago
Depends on your definition of “brute force”, I guess. Between explicitly deriving some algorithm from an understanding of causal relationships, v.s. training a neural net on billions of samples without encoding any broader understanding of the world, the latter approach is the one I would have called “brute force”.
xondono 5y ago
Using NN for cache management is not that new, and is a simpler approach in comparison to Intels as far as I know.

IgorPartola 5y ago

At 192MB, just give me explicit access to this cache as memory and I’ll decide what goes into it. No need to try to predict what my code will want, I can just tell you.

derefr 5y ago
Imagine coding for a 64-bit version of the 6502, where you have a 192MB "zero page" :)
BBC-vs-neolibs 5y ago
If you boot on actual hardware (no VM), if your application code + kernel (or unikernel) fits in under 192 megabytes, you are golden.
VHRanger 5y ago
Put everything in contiguous memory and load it, it'll be in cache
- IgorPartola 5y ago
  That not a great solution. My working data set is likely larger than 192MB. But my application is much more likely to know what data is in the hot path than the CPU is to guess it. I might want to put my DB index into this cache, or I might want to put only a part of it. I might want to preload the data then work on it for a bit, then load the next chunk. I might want one core preparing the data in RAM while another core is getting ready to work on it in this cache. Essentially I want full access to it independent of the main RAM because I can do a lot more with that. Think of this as the difference between RAM and disk: is RAM only useful as a cache to what’s on disk?
  - vlovich123 5y ago
    We know both positions are correct in practice. Most automatic prediction/management algorithms perform better on 90% of the code that’s run on them. However, careful performance tuning over time by experts can outperform generic solutions, sometimes dramatically. I generally err on the side of “I’m not as smart as I think I am” in these kinds of discussions. It’s not that I can never do it. I’m sure if I studied it a bit I could. The reason is that 99% of the lines of code I’ve ever written don’t warrant that kind of attention. I’m sure there are uses like when you’re writing some core fundamental algorithm in the fields of compression, crypto algorithm, hash, video processing, etc etc etc. I don’t work in those spaces though so the benefit is much more marginal. It’s possible 192mb is enough to start needing some explicit memory management. It makes the coding model much more complex though. And in a greedy software system where your code isn’t the only one running, such complexity doesn’t necessarily net overall wins. It’s the reason we have drivers and OSes even though we started with each piece of content bundling explicit HW support (at much better perf generally)

bcrl 5y ago

Actually, this is where 3d stacking changes the game. With 1 additional vertical layer, you can get double the memory at practically the same latency, as the source of latency is related to the wire distance (and capacitance) in the 2d plane. Latency can be lowered with 3d stacking as well if you reduce the 2d area of the memory array. This is how the industry will keep scaling going when 2d feature sizes can no longer be shrunk.

davuinci 5y ago
What's the limitation on adding additional vertical layers though? One apparent limitation could be that by adding several vertical layers it would require the wire to be longer.
- cogman10 5y ago
  2 limitations. 1. Heat doesn't go away with layers and gets harder to cool. 2. These are still layers. You want each layer to be as flat as possible. As the layers deform you lose the ability to cleanly add more layers. Current lithography is very much based on the notion that everything is flat.
- klodolph 5y ago
  Heat dissipation, although the power consumption of SRAM is mostly affected by I/O rather than capacity.