5 ms·
Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
- greenail 5y ago> Model training is done with mixed precision on the NVIDIA DGX SuperPOD-based Selene supercomputer powered by 560 DGX A100 servers networked with HDR InfiniBand in a full fat tree configuration. Each DGX A100 has eight NVIDIA A100 80GB Tensor Core GPUs, fully connected to each other by NVLink and NVSwitch. A similar reference architecture is used by Microsoft for Azure NDv4 cloud supercomputers. That is a big machine, how much does that run per month?
- thuccess129 5y agoHow will that model compare to Tesla's Dojo D1 with GPUs taken offline? Can't wait to know the answer.