4 ms·
Sigh, I wish people wouldn’t say “petaflop” for these. https://www.nvidia.com/en-us/data-center/a100/ https://www.nvidia.com/en-us/data-center/a100/ is the mos
by boulos 6y ago
Sigh, I wish people wouldn’t say “petaflop” for these.
https://www.nvidia.com/en-us/data-center/a100/ https://www.nvidia.com/en-us/data-center/a100/ is the most official reference. If you scroll to the bottom, you’ll see that an A100 part can do ~20 Teraflops (either FP32 or FP64 in little-matrix aka tensor mode). When they say “each A100 can do 5 petaflops”, they mean each DGX which has 8 such cards and thus they mean 600-ish something-ops per card. The generous assumption is that’s FP16 or bfloat16 for sparse ops, and therefore they are “flops”.
The reality is that if someone says “supercomputer” in the general sense, they mean scientific computing and so mean a double-precision LINPACK benchmark. The 1120 A100 parts (8x140) doing 20 “real” teraflops each has an absolute peak of about 22 Petaflops (and older code without tensor mode would be half that on FP64).
tl;dr: ML isn’t scientific computing, and those are different flops, but “10 petaflops” just doesn’t sound as impressive.
- deleted 6y ago[deleted]
- rrss 6y agoYou're the "disclaimer: I work for google cloud" person, right? Maybe you can help solve this problem (and it's a real one, it massively sucks that everyone counts whatever "flops" they want). Start here: https://cloud.google.com/tpu https://cloud.google.com/tpu > Cloud TPU v3 Pod > 100+ petaflops Is the Cloud TPU v3 even functionally capable of computing in fp64 at any performance?
- boulos 6y agoYes, (and I considered including that disclaimer). I give them shit about that all the time. I'm an equal-opportunity complainer.
- rrss 6y agoI'm glad to hear it. but it's sorta disheartening that whoever makes these decisions apparently don't care. Marketing going to market, I guess.
- coolgeek 6y ago> Marketing going to market This is one of those situations where the vulgate enhances comprehension - here eliminating ambiguity: Marketing gonna market
- fizixer 6y agoI agree with you as well as with the response to your comment, in that both nVidia and Google are engaging in misleading claims about their TPUs. But I'd like to add two points: - I don't think they can get away with it when it comes to Top-500 rankings. Those rankings are based on LINPACK scores and this supercomputer would end up not scoring high enough and will be placed in the right spot in the list. So it's not a big concern to me. - "ML isn't scientific computing" goes both ways. Sure tensor-tera-ops are not teraflops, they're specific to operations involving artificial neural networks (ANNs). But when you take the claims from the past about how much computing power it'll take for rivaling that of a human brain, and folks came up with 100 petaflops, or 1 exaflop. Well those should not be called "f"lops either. Because when it comes to brain inspired computing, ANN tensor-tera-ops are a more reliable number than FP32 or FP64. And if it's 100 peta-ops of ANN compute, well we're already past that, and can easily create a supercomputer with 1 tensor-exa-ops. And that means we have reached the hardware capacity required to emulate human-level intelligence in a machine (i.e., the only thing missing is the right set of algorithms). Conclusion: Tensor-tera-ops are not FP ops and should not be used for placement in the Top-500 list. But tensor-tera-ops have enough significance that it warrants creating a new list Top-500-Tensor and ranking Tensor-supercomputers on that list.
- deleted 6y ago[deleted]
- fluffy87 6y agoFuly agree - TL;DR pick the right benchmark. If you are doing AI the Top500,is meaningless and if you are doing classical HPC these ExaFlop int8 supercomputers are meaningless
- Todd 6y agoMaybe we need a new unit of measure: the flip. Similar to how SN came up with the bi suffixes for binary (mebi vs. mega, etc.). The flip would be a small flop.