4 ms·
250
by jleahy 3y ago
250
- metadat 3y agoInteresting, so what is the compute power of the 1000-node A100 super cluster my team has been allocated at work? I was expecting Google to be much bigger than us.
- sanxiyn 3y agoThis is for Google Cloud users. My understanding is that Google mostly uses TPU internally.
- Aissen 3y agoBack of the envelope math is that H100 is twice as fast as A100 (task may vary). So your 1000-node A100 very, very fast. Now, the GPU-to-GPU links (NVLink) might often give them a big advantage for some workloads, letting them exchange data without going through the CPU, and virtually address more memory if your want to manipulate very large models. So it's hard to answer properly without knowing the topology of your cluster. Also, note that this "supercomputer", is probably "just" a DGX H100 in Google's DC.
- lwkl 3y agoThey use their own TPUs like described in this paper [0]. They talk about 4096-chip supercomputers so this should give you an idea about what we are talking here. The paper is pretty fascinating stuff. They are using optical interconnects for example, which sounded like science fiction a few years ago. [0] https://arxiv.org/pdf/2304.01433.pdf https://arxiv.org/pdf/2304.01433.pdf