4 ms·
1 H100 is only 80GB of HBM. I guess you mean a server with 4xH100 is 1 node?
by kristianp 3y ago
1 H100 is only 80GB of HBM. I guess you mean a server with 4xH100 is 1 node?
- karmasimida 3y agothis is essentially 400b params. With FP8, comparing to Grok'3 320B model, which requires 320GB VRam in int4, I think what the OP meant is actually 8 H100. Which is ... a lot to say the least. And all optimization is for latency, not throughput, because with 8 H100, you can easily hosted 4 replicas of 70B.
- kristianp 3y agoThanks for the correction, there are indeed 8x nodes. https://developer.nvidia.com/blog/introducing-nvidia-hgx-h100-an-accelerated-server-platform-for-ai-and-high-performance-computing/ https://developer.nvidia.com/blog/introducing-nvidia-hgx-h10...