Traditionally the HPC community means double precision when they talk about FLops, which consumer GPUs are not super great at (somewhere around 400GFlop or the

Traditionally the HPC community means double precision when they talk about FLops, which consumer GPUs are not super great at (somewhere around 400GFlop or the GTX1080 Ti). The V100 is better at 7 or so TFlops, but also an order of magnitude more expensive. The smaller of the two machines in the article (Cori) is a ~20 PF (Float64) scale machine. However, even then it is quite challenging to get anything close to peak performance out of these systems (a bit easier if you're doing DL will is more amenable to hand-optimized vendor libraries). Of course for DL, people routinely use reduced precision as you have alluded to. There, we're looking at exascale at the moment. The second machine mentioned in the article (Summit) has ~3EF of Float16 performance (of which their application reached ~1EF). For comparison (and assuming about 50% MXU utilization - which is about the maximum I've seen), you'd need about 200 TPUv2 pods (at $384/hour/pod for the general public) to reach the same scale on this problem. Now Google does have that scale, but it's certainly non-trival. Half-precision petascale is fairly routine these days. You can get that on the cloud for < $100/hr and I do routinely spin up such systems for ML training. However, petascale fp32/fp64 and exascale (b)f16 as discussed in the article are still fairly rare and usually preceded by months of planning to make sure things go right and the computer power is used usefully. Disclaimer: not involved in the work, but working closely with the folks who are and I've used both the mentioned systems myself.