3 ms·
“The 2 Exaflops Aurora supercomputer is a beast of a machine and the system will be used to power the Aurora genAI AI model.” and instead of a data center with
by yieldcrv 3y ago
“The 2 Exaflops Aurora supercomputer is a beast of a machine and the system will be used to power the Aurora genAI AI model.”
and instead of a data center with a distributed load of Nvidia cards, they’re going to do 1 supercomputer? Are these guys intentionally stuck in 2007 or am I missing something
- whatshisface 3y agoRacks full of GPU nodes are called supercomputers. Supercomputers that ran a single instance of an OS kernel are from even farther back, maybe the last century.
- elcomet 3y agoIt's a datacenter, not a single machine obviously. All "supercomputers" are datacenter today. They can use Nvidia or other processing units, it doesn't really matter.
- mirekrusin 3y ago...which is equivalent to just 62x H100 GPUs from Nvidia?
- AnthonyMouse 3y agoOne H100 has the same memory bandwidth as the two CPUs in each of the compute nodes. But the supercomputer has >9000 compute nodes, so 62x H100 GPUs might fall short there. They also might not be measuring the same thing. H100 GPUs have "3026 TFLOPS": https://www.pny.com/nvidia-h100 https://www.pny.com/nvidia-h100 ...at FP8. 51 TFLOPS at FP32. Supercomputer lists typically use FP64.
- godelski 3y ago> and instead of a data center with a distributed load of Nvidia cards You're missing something. Most models are trained on supercomputers. You still use DDP, but you also use Slurm. A supercomputer is just a many node machine, but one where you REALLY care about interconnect. It's why the fabric is one of the big points here. I/O is generally your bottleneck in these systems. Each node should be as powerful of a system as possible because of this. A datacenter doesn't need a PB/s connection between machines. Your I/O probably isn't your bottleneck and so the tradeoff between PCIe and ethernet (infiniband or whatever) isn't a big deal. Usually because your processes are __independent__ (containers). In a supercomputer, your processes may be parallel, but that doesn't mean they're independent. You'll spend a lot of time writing code to be non-blocking but you're going to have to map reduce at some point.
- HybridCurve 3y agoAurora Supercomputer: - 200Gb dragonfly interconnect fabric - 10,000 nodes with 2xCPU + 6xGPU with unified memory - 10PB of agreggate RAM - 230PB of storage Scale matters in supercomputing. Keeping latency low between everything allows you to solve different classes of problems that gets no speed up benefit by simply offloading compute to nvidia cards. It makes a huge difference how everything is connected.