5 ms·
No, those national lab supercomputers typically quote double-precision flops, whereas this appears to be about float16 tensor ops for inner product (they call i
by jhj 6y ago
No, those national lab supercomputers typically quote double-precision flops, whereas this appears to be about float16 tensor ops for inner product (they call it "AI flops").
The A100 is 9.7 general purpose Tflops of float64, versus 624 Tflops of float16 / bfloat16 (perhaps also just tensor cores not general purpose)?
So quoting it the same way, this Florida machine would only be a (700 * (9.7 / 624)) = 10.9 Pflop supercomputer, not counting host CPUs.
Granted though, the hot new HPC approach is to approximate certain expensive high-precision calculations with ML inference instead, but that only works for certain algorithms.
- boulos 6y agoJinx :). I really am surprised the marketing folks didn’t claim this was an ~3 exa-op supercomputer based on int4 w/ sparsity.
- fluffything 6y agoThe FLOPs quoted by national lab supercomputers are usually measured double-precision FLOPs from HPL (High-performance Linpack - at least on the Top500 reports). IIUC, HPL can use double-precision tensor operations. It might also be able to use mixed-precision for some computations. Are the performance numbers for this HPC system at University of Florida also from HPL ?
- jabl 6y ago> IIUC, HPL can use double-precision tensor operations. Lets put it this way: As of the A100 GPU generation, the tensor cores are able to do double precision calculations too. By using the tensor core the peak FP64 flops double to 19.5 Tflops. And yes, the CUDA BLAS library (that the HPL benchmark uses) uses the tensor cores on the A100. > It might also be able to use mixed-precision for some computations. There is currently a lot of interest in mixed precision algorithms, but I'm not sure that the algorithm used for HPL is amenable to that approach, nor whether it's allowed per the benchmark rules.
- jabl 6y agoThe machine will be 140 nodes, which is half the size of the current #7 on the top500 list which is otherwise identical: https://top500.org/system/179842/ https://top500.org/system/179842/ That #7 machine got 27.6 Pflops on HPL, so I would guess the Florida machine will get roughly half of that. (Please keep in mind the tensor cores on the A100 can do FP64, so the theoretical peak flops per GPU is 19.5)
- mkoubaa 6y agobesides, the next generation of cray (Shasta) provides an on prem cloud environment where you can get 1000 computers or one supercomputer depending on your needs. I think that's a better deal than a supercomputer that can't do anything else. one day when disaggregated hardware becomes common in data centers the idea of a supercomputer will seem dated