3 ms·

Samsung Debuts zHBM Prototype, Stacking Memory Directly on AI Accelerators

peter_d_sherman 1mo ago
>"According to the company [Samsung], zHBM can deliver up to eight times the data-processing performance of eighth-generation High Bandwidth Memory (HBM5) while improving performance per watt by a factor of three. Thermal resistance is reduced by more than half, enhancing both system stability and energy efficiency."
senectus1 1mo ago
if all thats true, this is some impressive specs.
tancop 1mo ago
Where is the HBM controller in this design? Stacked in a middle layer or integrated into the main GPU die?
eis 1mo ago
I know consumers hate the situation with ram and storage prices right now, as do I. But at least on the bright side all this AI investment has unlocked a lot of progress in a space that didn't see huge advancements in a good while. All these 10-20% improvements gen-on-gen have resulted in upgrade cycles of well over 5 years for many use cases in order to really feel like it's worth it. RAM capacities especially have felt near stagnant for a decade.
- literalAardvark 1mo ago
  The progress is nice, but HBM appears to be impossible to reuse so... Kind of wasteful. Chips at least had some aftermarket life in them.
marsx-dev 1mo ago
How much of the benefit here comes from shorter data paths, versus simply being able to fit more memory closer to the accelerator?
vkaku 1mo ago
A lot of the AI progress is stalled because these chips do not handle thermal contraction/expansion well at all, and end up permanently destroying these chips. HBM has been a failure at this scale with chips getting destroyed every 4 months or so during operations. I'd like to see some actual science saying, here was the problem, here's how we solved it, here's the AFR data, here's this running after X cycles etc. Nobody has done this reliably yet. That entire industry is hiding the bodies.
- _joel 1mo ago
  How many contraction/expansion cylces do you normally see in a DC setting, generally? Is there a measure for that to baseline against?
  - bearjaws 1mo ago
    IIRC it's thousands per day on these systems, mainly due to high power density and low mass, even small lapses in computation (100-500ms) rapidly change the temperatures of stacked die. So even a GPU averaging 98% utilization may have thousands of cycles per day. Compared to a regular server blade it may be dozens or barely any at all.
    - venussnatch 1mo ago
      Naive question, but could this not be fixed by a scheduler? If it's only idle 2% of the time, give it busy work for that 2%
      - adgjlsfhk1 1mo ago
        the easier answer would be to remove the ability to idle
keanebean86 1mo ago
What happened to the idea of having tiny tubes running through chips that liquid would flow through? Cooling the inside of chips directly. Did it end up causing more problems than it solved?
quietraster 1mo ago
stacking memory right on the accelerator feels inevitable. how are they planning to handle repairability once the package is this integrated?
- kennywinker 1mo ago
  Just throw it out and buy a replacement. The ol’ environment can take another one for the team.

>"According to the company [Samsung], zHBM can deliver up to eight times the data-processing performance of eighth-generation High Bandwidth Memory (HBM5) while improving performance per watt by a factor of three. Thermal resistance is reduced by more than half, enhancing both system stability and energy efficiency."

senectus1 1mo ago

if all thats true, this is some impressive specs.

tancop 1mo ago

Where is the HBM controller in this design? Stacked in a middle layer or integrated into the main GPU die?

eis 1mo ago

I know consumers hate the situation with ram and storage prices right now, as do I. But at least on the bright side all this AI investment has unlocked a lot of progress in a space that didn't see huge advancements in a good while. All these 10-20% improvements gen-on-gen have resulted in upgrade cycles of well over 5 years for many use cases in order to really feel like it's worth it. RAM capacities especially have felt near stagnant for a decade.

literalAardvark 1mo ago
The progress is nice, but HBM appears to be impossible to reuse so... Kind of wasteful. Chips at least had some aftermarket life in them.

marsx-dev 1mo ago

How much of the benefit here comes from shorter data paths, versus simply being able to fit more memory closer to the accelerator?

vkaku 1mo ago

A lot of the AI progress is stalled because these chips do not handle thermal contraction/expansion well at all, and end up permanently destroying these chips. HBM has been a failure at this scale with chips getting destroyed every 4 months or so during operations. I'd like to see some actual science saying, here was the problem, here's how we solved it, here's the AFR data, here's this running after X cycles etc. Nobody has done this reliably yet. That entire industry is hiding the bodies.

_joel 1mo ago
How many contraction/expansion cylces do you normally see in a DC setting, generally? Is there a measure for that to baseline against?
- bearjaws 1mo ago
  IIRC it's thousands per day on these systems, mainly due to high power density and low mass, even small lapses in computation (100-500ms) rapidly change the temperatures of stacked die. So even a GPU averaging 98% utilization may have thousands of cycles per day. Compared to a regular server blade it may be dozens or barely any at all.
  - venussnatch 1mo ago
    Naive question, but could this not be fixed by a scheduler? If it's only idle 2% of the time, give it busy work for that 2%
    - adgjlsfhk1 1mo ago
      the easier answer would be to remove the ability to idle

keanebean86 1mo ago

What happened to the idea of having tiny tubes running through chips that liquid would flow through? Cooling the inside of chips directly. Did it end up causing more problems than it solved?

quietraster 1mo ago

stacking memory right on the accelerator feels inevitable. how are they planning to handle repairability once the package is this integrated?

kennywinker 1mo ago
Just throw it out and buy a replacement. The ol’ environment can take another one for the team.