4 ms·

The H200 GPU die is the same as the H100, but its using a full set of faster 24GB memory stacks: https://www.anandtech.com/show/21136/nvidia-at-sc23-h200-accel

The H200 GPU die is the same as the H100, but its using a full set of faster 24GB memory stacks: https://www.anandtech.com/show/21136/nvidia-at-sc23-h200-accelerator-with-hbm3e-and-jupiter-supercomputer-for-2024 https://www.anandtech.com/show/21136/nvidia-at-sc23-h200-acc... This is an H100 141GB, not new silicon like the Nvidia page might lead one to believe.

latchkey 3y ago
What happened to the H100 NVL? https://www.anandtech.com/show/18780/nvidia-announces-h100-nvl-max-memory-server-card-for-large-language-models https://www.anandtech.com/show/18780/nvidia-announces-h100-n...
- brucethemoose2 3y ago
  I dunno. But thats a dual GPU product, so its not really 180GB.
  - az226 3y ago
    They are also doing a custom 4 chip Hopper GPU for a European HPC initiative. Seems like a good move. Basically allows you to have 4 H100s connected via NVLink without needing a separate SXM board.
    - latchkey 3y ago
      I'm holding out for 8 x MI300x on a single board. =)
- jauntywundrkind 3y ago
  This is a single-chip H100 NVL. Both are GH100's with the same tweaked 20% wider 6144-bit HBM3e (versus 5120 bit on other H100's) running at a higher speed. The HBM3e loadout is slightly different than H100 NVL's was going to be, but this definitely seems like a higher bin H100. It's basically as-if AMD had shipped a 7900 XT, then latter started selling the 7900 XTX; same chip, but they brought up all the memory controllers on this one.
sva_ 3y ago
It is remarkable how much GPU compute is limited by memory speed.
- PartiallyTyped 3y ago
  In many cases it's the same for CPU. AMD's cpus with bigger caches due to on-die 3D stacked memory are in a different league when it comes to perf.
- brucethemoose2 3y ago
  Depends on the workload. Sometimes things really are compute bound, and sometimes you get a "big" workload that still fits nicely in the GPU's L2. Generative AI is mostly at the far end of "memory bound." Some ML startups (like Graphcore) seemed to bet on large caches, sparsity and clever preprocessing instead of raw memory bandwidth, but I think their strategy was compromised when model sizes exploded. Even Cerebras was kinda caught off guard when their 40GB pizza was suddenly kind of cramped.
  - zozbot234 3y ago
    Current ML architectures tend to be heavily optimized for ease of very large scale parallelism in training, even at the expense of a bigger model size and compute cost. So there may be some hope for different architectures as we stop treating idle GPUs as being basically available for free and start budgeting more strictly for what we use.
  - tbalsam 3y ago
    In hlb-CIFAR10 the MaxPooling ops are the slowest kernels now and take longer than all 7 of the convolution operations combined, if I understand correctly. Memory-bound operations seem to rather consistently be the limiting factor in my personal ML research work, at least. It can be rather annoying!
- westurner 3y ago
  What would make [HBM3E] GPU memory faster? High Bandwidth Memory > HBM3E: https://en.wikipedia.org/wiki/High_Bandwidth_Memory#HBM3E https://en.wikipedia.org/wiki/High_Bandwidth_Memory#HBM3E
WhitneyLand 3y ago
For anyone wondering how this applies to big LLMs, 144GB is big, but you’d need to roughly double this to train Gpt 3.x fitting everything in memory at once. Of course even If 300GB GPUs were available tomorrow, and you sold a million house to buy as many as that would allow it’d still take years to train once.

latchkey 3y ago

What happened to the H100 NVL? https://www.anandtech.com/show/18780/nvidia-announces-h100-nvl-max-memory-server-card-for-large-language-models https://www.anandtech.com/show/18780/nvidia-announces-h100-n...

brucethemoose2 3y ago
I dunno. But thats a dual GPU product, so its not really 180GB.
- az226 3y ago
  They are also doing a custom 4 chip Hopper GPU for a European HPC initiative. Seems like a good move. Basically allows you to have 4 H100s connected via NVLink without needing a separate SXM board.
  - latchkey 3y ago
    I'm holding out for 8 x MI300x on a single board. =)
jauntywundrkind 3y ago
This is a single-chip H100 NVL. Both are GH100's with the same tweaked 20% wider 6144-bit HBM3e (versus 5120 bit on other H100's) running at a higher speed. The HBM3e loadout is slightly different than H100 NVL's was going to be, but this definitely seems like a higher bin H100. It's basically as-if AMD had shipped a 7900 XT, then latter started selling the 7900 XTX; same chip, but they brought up all the memory controllers on this one.

sva_ 3y ago

It is remarkable how much GPU compute is limited by memory speed.

PartiallyTyped 3y ago
In many cases it's the same for CPU. AMD's cpus with bigger caches due to on-die 3D stacked memory are in a different league when it comes to perf.
brucethemoose2 3y ago
Depends on the workload. Sometimes things really are compute bound, and sometimes you get a "big" workload that still fits nicely in the GPU's L2. Generative AI is mostly at the far end of "memory bound." Some ML startups (like Graphcore) seemed to bet on large caches, sparsity and clever preprocessing instead of raw memory bandwidth, but I think their strategy was compromised when model sizes exploded. Even Cerebras was kinda caught off guard when their 40GB pizza was suddenly kind of cramped.
- zozbot234 3y ago
  Current ML architectures tend to be heavily optimized for ease of very large scale parallelism in training, even at the expense of a bigger model size and compute cost. So there may be some hope for different architectures as we stop treating idle GPUs as being basically available for free and start budgeting more strictly for what we use.
- tbalsam 3y ago
  In hlb-CIFAR10 the MaxPooling ops are the slowest kernels now and take longer than all 7 of the convolution operations combined, if I understand correctly. Memory-bound operations seem to rather consistently be the limiting factor in my personal ML research work, at least. It can be rather annoying!
westurner 3y ago
What would make [HBM3E] GPU memory faster? High Bandwidth Memory > HBM3E: https://en.wikipedia.org/wiki/High_Bandwidth_Memory#HBM3E https://en.wikipedia.org/wiki/High_Bandwidth_Memory#HBM3E

WhitneyLand 3y ago

For anyone wondering how this applies to big LLMs, 144GB is big, but you’d need to roughly double this to train Gpt 3.x fitting everything in memory at once. Of course even If 300GB GPUs were available tomorrow, and you sold a million house to buy as many as that would allow it’d still take years to train once.