3 ms·
I feel like a lot of the big GPU clusters that companies like Meta are using would score highly on the benchmark, even though Linpack seems to be based on fp64
by porphyra 2y ago
I feel like a lot of the big GPU clusters that companies like Meta are using would score highly on the benchmark, even though Linpack seems to be based on fp64 which the H200 is kinda sucky at. However, there's only one GH200-based machine (Alps, with 2688 GH200 nodes) in the top 10 even though I'm fairly sure it's quite a bit smaller than whatever Meta is training Llama 3 on. Is there any reason why we don't see those show up in TOP500?
- pixl97 2y agoThey are busy running training/inference on them and haven't benchmarked them?
- porphyra 2y agoSo these benchmark lists like TOP500 are just fundamentally flawed because the serious players are too busy doing actual work than to participate?
- pixl97 2y agoI mean there is no rule if you buy a supercomputer that you have to benchmark it and submit the results. This said, in the days before AI the number and type of players that had this amount of compute were also commonly the types to submit their scores to said benchmark lists.
- CaliforniaKarl 2y agoMost players that had this amount of compute also tend to pay less than the equivalent corporate wage, and so besides being a good way to stress-test your cluster (prior to general availability), it gives you the positive feeling of “I helped make this, I help run this.”
- kkielhofner 2y agoI work with a couple of the national labs and the talent there is generally incredible. "I went to school for > 25 years so my life's work could be selling more ads" isn't a motivator for most of these people.
- CaliforniaKarl 2y agoThanks much for the info! Sentiments like that make me hopeful for humanity.
- bee_rider 2y agoIt is a good advertisement for interconnect vendors.
- adrian_b 2y agoTOP500 and other similar HPC benchmark lists are only for FP64 computations. While both NVIDIA and AMD design their top GPU model for both FP64 and AI/ML workloads, to save on the design cost, you can do AI training using GPUs that have only high AI performance (like RTX 4090 or its workstation counterpart, RTX 6000), without implementing FP64 operations at all (the FP64 performance of RTX 4090 is negligible, being worse than of any decent cheap CPU, it is provided only for compatibility in testing).
- davidmr 2y agoOr have them set up in a way that makes them hard to run full-system benchmarks on. I can think of a couple of financial firms that have clusters that would rate on the Top 10, but a) as you suspected, they’re too busy running the money printers to take them down for a few days to run benchmarks, and b) they’re split up into more manageable little clusters, so the high speed fabrics don’t connect and allow every node to talk to every other node, which you need for an HPL run.
- hnaccount_rng 2y agoAre you willing to dump a name or two of those finance firms?
- alephnerd 2y ago> I feel like a lot of the big GPU clusters that companies like Meta are using would score highly Some might, but most of the work done on GPU compute was subsidized by DoE Exascale projects like Aurora, like my anecdote about the DGX product line above.
- brrrrrm 2y agoA single 8xH100 node hits 15.6 fp16 petaFlops
- dekhn 2y agoThe machines that the large ML companies are using are typically not going to run TOP500 competitively. They tend to skip the MPI stack, and weren't optimized for LINPACK performance. When I worked on TPUs there was no interest in attempting to benchmark them using supercomputer workloads (probably a good thing).