5 ms·

Much more interested in architecture design and memory hierarchy than flops. Anything interesting going on in caching or memory hardware? All the problems I wo

Much more interested in architecture design and memory hierarchy than flops. Anything interesting going on in caching or memory hardware? All the problems I work on need benefit from memory bandwidth and cache latency than raw FLOPS. I imagine others are in the same boat. I was hoping this would be the start of some more architecture diversity like apples tile based deferred rendering.

winter_blue 6y ago
In terms of memory architecture -- I've heard that memory is a bottleneck for the GPU, specifically the time it takes to move stuff from RAM (main memory) to the GPU's RAM/memory. If it is such a big bottleneck, then why don't we (yet) see powerful GPU sharing the same die as the CPU and accessing/using/sharing RAM with the CPU (like integrated GPUs do)? Then there'd be a zero bottleneck. You would just load whatever into RAM, and just give the (super-powerful integrated) GPU a pointer/address. Bam, done. Why hasn't this happened yet? / What am I missing/misunderstanding here?
- wolf550e 6y ago
  You would have 600W in a single socket.
- tarlinian 6y ago
  Mainly because what you're saying isn't true for many workloads? (Also this already works for most existing integrated GPUs.) The types of workloads run on GPUs typically like very high memory bandwidth and are usually willing to live with higher memory latency to get it. Onboard GPU memory is usually built with this in mind (trade off capacity and latency for increases in bandwidth). This is generally speaking the opposite of what you want in a CPU where people often want very high memory capacity and lower latencies, but may not limited by memory bandwidth, so simply sticking a GPU on die and giving it access to a memory subsystem that was not designed to feed a GPU is not going to make anything better.
- qzw 6y ago
  Besides the massive heat/power issues you would have, main memory like DDR4 is optimized for different characteristics than graphics memory like GDDR5 (mostly trade offs in latency vs bandwidth).
- nordsieck 6y ago
  > I've heard that memory is a bottleneck for the GPU, specifically the time it takes to move stuff from RAM (main memory) to the GPU's RAM/memory. If it is such a big bottleneck, then why don't we (yet) see powerful GPU sharing the same die as the CPU and accessing/using/sharing RAM with the CPU (like integrated GPUs do)? Then there'd be a zero bottleneck. You would just give load whatever into RAM, and just give the (super-powerful integrated) GPU a pointer/address. Bam, done. Why hasn't this happened yet? / What am I missing/misunderstanding here? That is not the only bottleneck involved. Historically, GPUs have used GDDR ram as opposed to general purpose DDR memory. One of the key differences between GDDR and DDR is the bus width, which can be as large as 1024 bits, compared to conventional ram with a 64 bit bus width (although dual channel is effectively 128 bits). This much wider bus results in much higher memory bandwidth which is generally necessary to feed the truly enormous number of functional units in a GPU. I suppose you could ask: why doesn't everyone just standardize on GDDR? 1. This would dramatically increase cache line size. I don't have data, but I assume this would generally be bad. 2. My recollection (but I don't have a source for this) is that DDR has lower latency than GDDR ram, so for branchy code (which CPUs often have to deal with, but GPUs typically never have to deal with), DDR could actually be faster. 3. DDR is cheaper to manufacture. Aside from being higher volume, a lower bus width just makes is simpler to manufacture.
  - PixelOfDeath 6y ago
    > 1. This would dramatically increase cache line size. I don't have data, but I assume this would generally be bad. Why would it change cache line size? GPUs also use cache lines in the range of 32-128 byte?! I think that is independent of the bus system/width.
    - nordsieck 6y ago
      > Why would it change cache line size? GPUs also use cache lines in the range of 32-128 byte?! I think that is independent of the bus system/width. I just assumed that bus width = cache line size. I guess I was wrong. Sorry.
  - jandrese 6y ago
- PixelOfDeath 6y ago
  GPUs can hide memory latency very well, because they are basically SMT on steroids. (Imagine instead of 2 threads per "core" you have 32-64 threads) But they are starved of memory bandwidth! And the lower latency memory CPUs prefer is not the same as the high bandwidth memory that GPUs like. Also there is different kind of caching. Modern APUs often have two ways to access memory. Over there own cache, or over the CPUs cache. So shared memory for CPU<->GPU gets the full advantage of a cache, but still it is a trade off. If you want some work done by the GPU part of an APU, by sharing a pointer, you can do that today. But from the point of view of the CPU there is no prediction beyond the "GPU do X" commands. And a very high latency until the job is done. So you need a minimum GPU job size for it to make sense.
- boxfire 6y ago
  The intel Kaby Lake G processors are Radeon RX Vega M GL/GH paired on-die with an 8th gen core series processor over an internal PCIE 3 x8 link. Seems like they could do much better, but its the only such product I know of. https://fuse.wikichip.org/news/1634/hot-chips-30-intel-kaby-lake-g/ https://fuse.wikichip.org/news/1634/hot-chips-30-intel-kaby-...
- formerly_proven 6y ago
  It's more of a bandwidth problem. The arithmetic intensity of a non-HBM GPU is around 100, meaning the GPU can perform 100 FLOPs for every float load/store.
- brandmeyer 6y ago
  > What am I missing/misunderstanding here Some (not all) GPU memory types are not cache coherent with the CPU. Some of the cache-coherent cases have poorer performance relative to the non-coherent memory from the perspective of the GPU's memory bandwidth and access latency.
- aspaceman 6y ago
  That's one bottleneck, but you're missing another bottleneck entirely. In many problems, we load the entirety of the data into the GPU's RAM anyways (some games do so with the entirety of their assets during load times). It doesn't matter that the data is already in RAM, because we care about how much data can come from GDDR to registers (in GB/s). You have to get the data to your compute units. This is your memory bandwidth. GDDR RAM wants to be accessed in bulk - large rows at once. Things are easy when each thread wants a subsequent byte, but if not things become much slower. Caching and other techniques can help mitigate this, and they're (imo) the place for a lot of creativity in architecture design. Having more potential FLOPS just means more ALUs.
  - winter_blue 6y ago
    Ah, things makes a lot more sense now. Thank you for explaining it so clearly.
- nwallin 6y ago
  What you're describing actually exists, it's called Heterogeneous Systems Architecture. https://en.wikipedia.org/wiki/Heterogeneous_System_Architecture https://en.wikipedia.org/wiki/Heterogeneous_System_Architect... It's nifty, but it's generally only used as a cost saving measure, not a performance one. It does, as you mention, broaden the scope of what is a useful target for GPU compute operations. Memory bandwidth is typically the bottleneck in GPUs. Meanwhile, the access patterns are typically very predictable. So they're able to prefetch data, so latency is generally not a problem. So GPU memory is designed to have very high bandwidth, even if it means completely tanking the latency. On the other hand, CPUs typically always need better memory latency, and most workloads do not saturate the memory bandwidth. Unlike in a GPU, many memory access patterns are unpredictable. The bottleneck of operations that are pointer heavy tends to be limited by latency. All tree operations and linked list iteration tend to be latency limited. Languages like C#, Java, Python and Javascript where all data lives behind a pointer tend to benefit significantly from improving latency. So while improvement memory bandwidth up to a point is important, there's much more attention given to latency.
jra101 6y ago
Tile based Deferred Rendering came from PowerVR. Prior to moving to their own GPU architecture, Apple licensed various PowerVR designs.
- ksec 6y ago
  Yes and it goes as far back as Saga Dreamcast. And the technique itself is now used in pretty much all of the current GPU from Nvidia to AMD and ARM. Still a little pissed at how Apple handled IMG / PowerVR.
  - Veedrac 6y ago
    NVIDIA uses tiled rasterization, but not TBDR. I believe the same is true of AMD.
  - monocasa 6y ago
    It goes farther back. You could buy PowerVR discrete GPUs for desktops far back as 1996, IIRC.
  - TomVDB 6y ago
    Neither AMD nor Nvidia are using deferred rendering.
ivalm 6y ago
If anything this is a move to maxwell-like architecture away from normal intel stuff. I feel like all GPU architectures are kind of converging (amd navi is also a move towards more maxwell-like arch)

winter_blue 6y ago

In terms of memory architecture -- I've heard that memory is a bottleneck for the GPU, specifically the time it takes to move stuff from RAM (main memory) to the GPU's RAM/memory. If it is such a big bottleneck, then why don't we (yet) see powerful GPU sharing the same die as the CPU and accessing/using/sharing RAM with the CPU (like integrated GPUs do)? Then there'd be a zero bottleneck. You would just load whatever into RAM, and just give the (super-powerful integrated) GPU a pointer/address. Bam, done. Why hasn't this happened yet? / What am I missing/misunderstanding here?

wolf550e 6y ago
You would have 600W in a single socket.
tarlinian 6y ago
Mainly because what you're saying isn't true for many workloads? (Also this already works for most existing integrated GPUs.) The types of workloads run on GPUs typically like very high memory bandwidth and are usually willing to live with higher memory latency to get it. Onboard GPU memory is usually built with this in mind (trade off capacity and latency for increases in bandwidth). This is generally speaking the opposite of what you want in a CPU where people often want very high memory capacity and lower latencies, but may not limited by memory bandwidth, so simply sticking a GPU on die and giving it access to a memory subsystem that was not designed to feed a GPU is not going to make anything better.
qzw 6y ago
Besides the massive heat/power issues you would have, main memory like DDR4 is optimized for different characteristics than graphics memory like GDDR5 (mostly trade offs in latency vs bandwidth).
nordsieck 6y ago
> I've heard that memory is a bottleneck for the GPU, specifically the time it takes to move stuff from RAM (main memory) to the GPU's RAM/memory. If it is such a big bottleneck, then why don't we (yet) see powerful GPU sharing the same die as the CPU and accessing/using/sharing RAM with the CPU (like integrated GPUs do)? Then there'd be a zero bottleneck. You would just give load whatever into RAM, and just give the (super-powerful integrated) GPU a pointer/address. Bam, done. Why hasn't this happened yet? / What am I missing/misunderstanding here? That is not the only bottleneck involved. Historically, GPUs have used GDDR ram as opposed to general purpose DDR memory. One of the key differences between GDDR and DDR is the bus width, which can be as large as 1024 bits, compared to conventional ram with a 64 bit bus width (although dual channel is effectively 128 bits). This much wider bus results in much higher memory bandwidth which is generally necessary to feed the truly enormous number of functional units in a GPU. I suppose you could ask: why doesn't everyone just standardize on GDDR? 1. This would dramatically increase cache line size. I don't have data, but I assume this would generally be bad. 2. My recollection (but I don't have a source for this) is that DDR has lower latency than GDDR ram, so for branchy code (which CPUs often have to deal with, but GPUs typically never have to deal with), DDR could actually be faster. 3. DDR is cheaper to manufacture. Aside from being higher volume, a lower bus width just makes is simpler to manufacture.
- PixelOfDeath 6y ago
  > 1. This would dramatically increase cache line size. I don't have data, but I assume this would generally be bad. Why would it change cache line size? GPUs also use cache lines in the range of 32-128 byte?! I think that is independent of the bus system/width.
  - nordsieck 6y ago
    > Why would it change cache line size? GPUs also use cache lines in the range of 32-128 byte?! I think that is independent of the bus system/width. I just assumed that bus width = cache line size. I guess I was wrong. Sorry.
- jandrese 6y ago
PixelOfDeath 6y ago
GPUs can hide memory latency very well, because they are basically SMT on steroids. (Imagine instead of 2 threads per "core" you have 32-64 threads) But they are starved of memory bandwidth! And the lower latency memory CPUs prefer is not the same as the high bandwidth memory that GPUs like. Also there is different kind of caching. Modern APUs often have two ways to access memory. Over there own cache, or over the CPUs cache. So shared memory for CPU<->GPU gets the full advantage of a cache, but still it is a trade off. If you want some work done by the GPU part of an APU, by sharing a pointer, you can do that today. But from the point of view of the CPU there is no prediction beyond the "GPU do X" commands. And a very high latency until the job is done. So you need a minimum GPU job size for it to make sense.
boxfire 6y ago
The intel Kaby Lake G processors are Radeon RX Vega M GL/GH paired on-die with an 8th gen core series processor over an internal PCIE 3 x8 link. Seems like they could do much better, but its the only such product I know of. https://fuse.wikichip.org/news/1634/hot-chips-30-intel-kaby-lake-g/ https://fuse.wikichip.org/news/1634/hot-chips-30-intel-kaby-...
formerly_proven 6y ago
It's more of a bandwidth problem. The arithmetic intensity of a non-HBM GPU is around 100, meaning the GPU can perform 100 FLOPs for every float load/store.
brandmeyer 6y ago
> What am I missing/misunderstanding here Some (not all) GPU memory types are not cache coherent with the CPU. Some of the cache-coherent cases have poorer performance relative to the non-coherent memory from the perspective of the GPU's memory bandwidth and access latency.
aspaceman 6y ago
That's one bottleneck, but you're missing another bottleneck entirely. In many problems, we load the entirety of the data into the GPU's RAM anyways (some games do so with the entirety of their assets during load times). It doesn't matter that the data is already in RAM, because we care about how much data can come from GDDR to registers (in GB/s). You have to get the data to your compute units. This is your memory bandwidth. GDDR RAM wants to be accessed in bulk - large rows at once. Things are easy when each thread wants a subsequent byte, but if not things become much slower. Caching and other techniques can help mitigate this, and they're (imo) the place for a lot of creativity in architecture design. Having more potential FLOPS just means more ALUs.
- winter_blue 6y ago
  Ah, things makes a lot more sense now. Thank you for explaining it so clearly.
nwallin 6y ago
What you're describing actually exists, it's called Heterogeneous Systems Architecture. https://en.wikipedia.org/wiki/Heterogeneous_System_Architecture https://en.wikipedia.org/wiki/Heterogeneous_System_Architect... It's nifty, but it's generally only used as a cost saving measure, not a performance one. It does, as you mention, broaden the scope of what is a useful target for GPU compute operations. Memory bandwidth is typically the bottleneck in GPUs. Meanwhile, the access patterns are typically very predictable. So they're able to prefetch data, so latency is generally not a problem. So GPU memory is designed to have very high bandwidth, even if it means completely tanking the latency. On the other hand, CPUs typically always need better memory latency, and most workloads do not saturate the memory bandwidth. Unlike in a GPU, many memory access patterns are unpredictable. The bottleneck of operations that are pointer heavy tends to be limited by latency. All tree operations and linked list iteration tend to be latency limited. Languages like C#, Java, Python and Javascript where all data lives behind a pointer tend to benefit significantly from improving latency. So while improvement memory bandwidth up to a point is important, there's much more attention given to latency.

jra101 6y ago

Tile based Deferred Rendering came from PowerVR. Prior to moving to their own GPU architecture, Apple licensed various PowerVR designs.

ksec 6y ago
Yes and it goes as far back as Saga Dreamcast. And the technique itself is now used in pretty much all of the current GPU from Nvidia to AMD and ARM. Still a little pissed at how Apple handled IMG / PowerVR.
- Veedrac 6y ago
  NVIDIA uses tiled rasterization, but not TBDR. I believe the same is true of AMD.
- monocasa 6y ago
  It goes farther back. You could buy PowerVR discrete GPUs for desktops far back as 1996, IIRC.
- TomVDB 6y ago
  Neither AMD nor Nvidia are using deferred rendering.

ivalm 6y ago

If anything this is a move to maxwell-like architecture away from normal intel stuff. I feel like all GPU architectures are kind of converging (amd navi is also a move towards more maxwell-like arch)