5 ms·

Question, why is DRAM still packed separately?

wmf 8y ago
A lot of reasons. It's physically quite large (multiple thousands of sq. mm. of silicon). Customers want different amounts of RAM so you could end up with a cartesian product explosion of SKUs. Customers want to upgrade RAM. System vendors and integrators don't want to give up more margins to the CPU.
extrapickles 8y ago
For higher power parts, heat dissipation is the primary reason. Most of the high end CPUs are purely heat limited in terms of performance. A TDP of 90W is about the limit for CPUs using cheap cooling techniques, and adding ram would lower that by 10-30W.
- ericd 8y ago
  If you interleaved the memory with the processing, wouldn’t it decrease the heat density, though, which afaik is the limiting factor? You can get rid of quite a lot more heat than that if it’s not in too small an area to conduct away quickly.
  - saltcured 8y ago
    The semiconductor processing for high-end DRAM is not the same as for high-end CPU logic or cache SRAM, so it is not cost-effective to put both on the same chip. This was a recurring obstable to past processing-in-memory R&D projects. To mix the two for thermal reasons would lead to some crazy multi-chip modules as in the "chiplet" approach reported recently. However, I suspect you would need far too many chiplets and too many bonds to really give you this thermal benefit rather than just having individual chiplets which are still hot spots with problematic cooling.
  - tmd83 8y ago
    I don't think that's feasible either. If you put memory in between chip element that means you now have to route around that adding to wastage. It also means the distance between the elements would increase slowing things down. At the speed that chips are running the physical distance matters (from what I recall).
  - mjevans 8y ago
    Processors also tend to run hot, while RAM is more reliable when cool. Much like some parts in human biology benefit from being outside of the core (enough to make it an evolutionary benefit even when it's quite an obvious defect due to vulnerability), having RAM in it's own cooling sub-domain makes sense, and since it's already out there the blade interconnect card design is space, cost, and cooling effective.
- nine_k 8y ago
  I wonder if the extra computing power achievable by more extensive cooling is worth it, in computation units per watt (including both system power and cooling power). I keep reading that some data centers switch to liquid cooling, so likely it does. Maybe with advances on this front, we'll see the increase of the CPU efficiency that the fabrication process can no more deliver.
stefan_ 8y ago
Why not? It's not like moving it to the same chip would make DRAM perform better, the signal lines are not the limiting factor there.
- wmf 8y ago
  HBM2 does have dramatically more bandwidth than DDR4.
  - monocasa 8y ago
    It has more bandwidth but about the same latency. CPUs fundamentally attack memory latency differently than GPUs do, and raw bandwidth is less important.
    - imtringued 8y ago
      It is extremely important when you have 64 cores orif you use wide SIMD instructions. Intel's Xeon Phi have 16GB HMC memory with 500GB/s bandwidth to prevent the CPUs from being starved by main memory. Imagine you have a cache miss on literally every memory access and have to pay a 100 nano second penalty to access main memory. SMT allows the CPU to switch to a different thread during a cache miss. Effectively you can have 128 memory loads at any given time. 100ns amortized over 128 threads is less than 1 nano second. Since the minimum size you can read is a 64 byte cache line this means that program will require at least 64GB/s of memory bandwidth even though your CPU is stuck on cache misses.
      - monocasa 8y ago
        That's what I meant by "CPUs fundamentally attack memory latency differently than GPUs do"; they rely on their relatively smart caches and prefetching logic to reduce bandwidth explosion from all of the cores, and the latency of accessing external DRAM. Of course if you thrash your cache on a modern general purpose CPU, you're going to have a really bad time. Pretty much if you're getting close to worse case memory access like in your example, you need to step back and re evaluate quite a few of your choices.
rwmj 8y ago
A different question is why is DRAM still colocated with the CPUs, and not centrally located over a very fast network? There was a company doing this (RNA Networks, founded 2006) which was sold to Dell and then apparently disappeared without trace. It might make more sense now that there are more types of memory-like storage which isn't quite as fast as DDR4, like Optane.
- wmf 8y ago
  The cost and latency of the very fast network is still too high, but for slower stuff Gen-Z is coming.
- walrus01 8y ago
  Latency between a CPU that has the memory controller built directly onto the die, and adjacent dram modules, is significantly less than if it is abstracted away somewhere else on the far side of a low latency, short distance Network link.
- nightcracker 8y ago
  The answer to that is very simple: latency. There is also an infinitely more tricky issue: security.
- uluyol 8y ago
  Attempts to do this are being made in academia. Some recent work tries to quantify the network requirements of this, and decides that it's far more feasible to use some local memory as a caches than to have everything be remote. Network requirements: https://www.usenix.org/conference/osdi16/technical-sessions/presentation/gao https://www.usenix.org/conference/osdi16/technical-sessions/... Two systems that use remote memory: https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/gu https://www.usenix.org/conference/nsdi17/technical-sessions/... https://www.usenix.org/conference/osdi18/presentation/shan https://www.usenix.org/conference/osdi18/presentation/shan
monocasa 8y ago
Smae reason that they split off the IO block here, but exaggerated. DRAM is very different from a process perspective, with very different constraints. Even eDRAM tends to be a compromise in a lot of ways that wouldn't be competitive if it weren't literally on the same chip as the logic. Additionally DRAM more or less has to be separate chips due to the sheer surface area for yield reasons. The amount of DRAM in a decent sized server is close to a whole wafer.

wmf 8y ago

A lot of reasons. It's physically quite large (multiple thousands of sq. mm. of silicon). Customers want different amounts of RAM so you could end up with a cartesian product explosion of SKUs. Customers want to upgrade RAM. System vendors and integrators don't want to give up more margins to the CPU.

extrapickles 8y ago

For higher power parts, heat dissipation is the primary reason. Most of the high end CPUs are purely heat limited in terms of performance. A TDP of 90W is about the limit for CPUs using cheap cooling techniques, and adding ram would lower that by 10-30W.

ericd 8y ago
If you interleaved the memory with the processing, wouldn’t it decrease the heat density, though, which afaik is the limiting factor? You can get rid of quite a lot more heat than that if it’s not in too small an area to conduct away quickly.
- saltcured 8y ago
  The semiconductor processing for high-end DRAM is not the same as for high-end CPU logic or cache SRAM, so it is not cost-effective to put both on the same chip. This was a recurring obstable to past processing-in-memory R&D projects. To mix the two for thermal reasons would lead to some crazy multi-chip modules as in the "chiplet" approach reported recently. However, I suspect you would need far too many chiplets and too many bonds to really give you this thermal benefit rather than just having individual chiplets which are still hot spots with problematic cooling.
- tmd83 8y ago
  I don't think that's feasible either. If you put memory in between chip element that means you now have to route around that adding to wastage. It also means the distance between the elements would increase slowing things down. At the speed that chips are running the physical distance matters (from what I recall).
- mjevans 8y ago
  Processors also tend to run hot, while RAM is more reliable when cool. Much like some parts in human biology benefit from being outside of the core (enough to make it an evolutionary benefit even when it's quite an obvious defect due to vulnerability), having RAM in it's own cooling sub-domain makes sense, and since it's already out there the blade interconnect card design is space, cost, and cooling effective.
nine_k 8y ago
I wonder if the extra computing power achievable by more extensive cooling is worth it, in computation units per watt (including both system power and cooling power). I keep reading that some data centers switch to liquid cooling, so likely it does. Maybe with advances on this front, we'll see the increase of the CPU efficiency that the fabrication process can no more deliver.

stefan_ 8y ago

Why not? It's not like moving it to the same chip would make DRAM perform better, the signal lines are not the limiting factor there.

wmf 8y ago
HBM2 does have dramatically more bandwidth than DDR4.
- monocasa 8y ago
  It has more bandwidth but about the same latency. CPUs fundamentally attack memory latency differently than GPUs do, and raw bandwidth is less important.
  - imtringued 8y ago
    It is extremely important when you have 64 cores orif you use wide SIMD instructions. Intel's Xeon Phi have 16GB HMC memory with 500GB/s bandwidth to prevent the CPUs from being starved by main memory. Imagine you have a cache miss on literally every memory access and have to pay a 100 nano second penalty to access main memory. SMT allows the CPU to switch to a different thread during a cache miss. Effectively you can have 128 memory loads at any given time. 100ns amortized over 128 threads is less than 1 nano second. Since the minimum size you can read is a 64 byte cache line this means that program will require at least 64GB/s of memory bandwidth even though your CPU is stuck on cache misses.
    - monocasa 8y ago
      That's what I meant by "CPUs fundamentally attack memory latency differently than GPUs do"; they rely on their relatively smart caches and prefetching logic to reduce bandwidth explosion from all of the cores, and the latency of accessing external DRAM. Of course if you thrash your cache on a modern general purpose CPU, you're going to have a really bad time. Pretty much if you're getting close to worse case memory access like in your example, you need to step back and re evaluate quite a few of your choices.

rwmj 8y ago

A different question is why is DRAM still colocated with the CPUs, and not centrally located over a very fast network? There was a company doing this (RNA Networks, founded 2006) which was sold to Dell and then apparently disappeared without trace. It might make more sense now that there are more types of memory-like storage which isn't quite as fast as DDR4, like Optane.

wmf 8y ago
The cost and latency of the very fast network is still too high, but for slower stuff Gen-Z is coming.
walrus01 8y ago
Latency between a CPU that has the memory controller built directly onto the die, and adjacent dram modules, is significantly less than if it is abstracted away somewhere else on the far side of a low latency, short distance Network link.
nightcracker 8y ago
The answer to that is very simple: latency. There is also an infinitely more tricky issue: security.
uluyol 8y ago
Attempts to do this are being made in academia. Some recent work tries to quantify the network requirements of this, and decides that it's far more feasible to use some local memory as a caches than to have everything be remote. Network requirements: https://www.usenix.org/conference/osdi16/technical-sessions/presentation/gao https://www.usenix.org/conference/osdi16/technical-sessions/... Two systems that use remote memory: https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/gu https://www.usenix.org/conference/nsdi17/technical-sessions/... https://www.usenix.org/conference/osdi18/presentation/shan https://www.usenix.org/conference/osdi18/presentation/shan

monocasa 8y ago

Smae reason that they split off the IO block here, but exaggerated. DRAM is very different from a process perspective, with very different constraints. Even eDRAM tends to be a compromise in a lot of ways that wouldn't be competitive if it weren't literally on the same chip as the logic. Additionally DRAM more or less has to be separate chips due to the sheer surface area for yield reasons. The amount of DRAM in a decent sized server is close to a whole wafer.