7 ms·
1 petaflop on a chip?? What is the catch?
by savant_penguin 5y ago
1 petaflop on a chip?? What is the catch?
- dragontamer 5y agoTensor petaflops are useful in only very few circumstances. One of which is the highly lucrative deep learning community though.
- cjbgkagh 5y agoThe main tensor op is a matmul intrinsic which is useful for way more than just deep learning. Edit; many of these speeds are low precision which is less useful outside of deep learning, but the higher precision matmul ops in the tensor cores are still very fast and very useful for wide variety of tasks.
- dragontamer 5y ago> but the higher precision matmul ops in the tensor cores are still very fast and very useful for wide variety of tasks. The FP64 matrix-multiplication is only 60 TFlops, no where near the advertized 1000 TFlops. TF32 matrix-multiplication is a poorly named 16-bit operation.
- cjbgkagh 5y agoYou are indeed correct, I was (kinda) fooled by the marketing and I think that TF32 is deceptively named. I think the tensor cores are being used in this architecture for FP64 and 60 TFlops is still pretty decent. I'm on Turing architecture so I've never used TF32. I've only used FP32 and FP16 but FP32 isn't supported by these tensor cores.
- bcatanzaro 5y agoWell the addition is done in FP32, and it's a 32-bit storage format in memory, so calling it a 16-bit format isn't right either. It's really a hybrid format where everything is 32-bit except multiplication. Given that it's 32-bit in memory (so all your data structures are 32-bit) and also that in my experience using it is very transparent (I haven't run into any numerical issues compared to full FP32), I think calling it a 32-bit format is a reasonable compromise.
- dragontamer 5y ago> Well the addition is done in FP32 Addition is done in 10-bit mantissa. So maybe TF19 might be the better name, since its a 19-bit format (slightly more than 16-bit BFloats). Really, its a BFloat with a 10-bit mantissa instead of a 7-bit mantissa. 10-bit mantissa matches FP16, while the 8-bit exponent matches FP32. So TF19 probably would have been the best name, but NVidia like marketing so they call it TF32 instead.
- bcatanzaro 5y agoIt's a 32-bit format in memory and the additions are done with 32-bits.
- dragontamer 5y agoI admit that I don't have the hardware to test your claims. But pretty much all the whitepapers I can find on TF32 explicitly state the 10-bit mantissa, suggesting that this is at best, a 19-bit format. 1-bit sign + 8-bit exponent + 10-bit mantissa. Yes, the system will read/write the 32-bit value to RAM. But if there's only 10-bits of mantissa in the circuits, you're only going to get 10-bits of precision (best case). The 10-bit mantissa makes sense because these systems have FP16 circuits (1 + 5-bit exponent + 10-bit mantissa) and BFloat16 circuits (1 sign + 8-bit exponent + 7-bit mantissa). So the 8-bit exponent circuit + 10-bit mantissa circuit exists physically on those NVidia cores. ------- But the 'Tensor Cores' do not support 32-bit (aka: 23-bit mantissa) or higher.
- my123 5y agoYup, in a semi-related field, NVIDIA has 3xTF32 for cases needing higher precision: https://github.com/NVIDIA/cutlass/discussions/361 https://github.com/NVIDIA/cutlass/discussions/361
- touisteur 5y agoThere's a paper on getting fp32 accuracy using tf32 tensor cores and losing 3x efficiency. Can't wait to try it with cutlass... once I get how to use cutlass, woof.
- aninteger 5y agoGiven that it's Nvidia, no Linux support. That's the catch.
- AHTERIX5000 5y agoNo Linux support? Guess I'll have to keep using Solaris with my A4000!
- ScaleneTriangle 5y agoI thought that only applied to their consumer products.
- TheRealSteel 5y agoDon't they provide Linux drivers for their gaming graphics cards too, just not open source?
- gpm 5y agoYes
- jsheard 5y agoTheir consumer products have Linux support too, the catch is just that the drivers are proprietary binary blobs
- simulate-me 5y agoNvidia provides Linux drivers for their server chips.
- TheRealSteel 5y agoDon't they provide them for their consumer cards too, just that it's a closed source binary blob?
- throw0101a 5y ago
- hughrr 5y ago700 watts so being NVidia it'll blow up in 6 months and you'll need to wait in a queue for 6 months to RMA it because all the miners had bought up the entire supply chain.
- touisteur 5y agoThose datacenter/hpc GPUs don't seem to get bought so much by the mining community? I don't have problems sourcing some through the usual channels (HPE, dell,...?). But you need somehow deep pockets.
- p1esk 5y agoThe catch is it's only for TF32 computations (Nvidia proprietary 19 bit floating point number format)
- cjbgkagh 5y agoI missed that, to me that makes the ‘32’ in the name misleading.
- p1esk 5y agoTF32 = FP32 range + FP16 precision
- cjbgkagh 5y agoWhy not call it TF19 then.
- p1esk 5y agoBecause your existing FP32 models should run fine when converted to TF32, so TF32 is equivalent to FP32 as far as DL practitioners are concerned.
- cjbgkagh 5y agoThere is a lot of redundancy in DL that forgives all manner of sins; think it’s sneaky.
- 37ef_ced3 5y agoBecause it's 32-bits wide in memory. The effective mantissa is like FP16 but it's padded out to be the same size as FP32. In other words, there's 1 sign bit, 8 exponent bits, 10 mantissa bits that are USED, and 13 mantissa bits that are IGNORED. 1 + 8 + 10 + 13 = 32 The 13 ignored mantissa bits are part of the memory image: they pad the number out to 32-bit alignment.
- 5y ago
- peter303 5y agoDP Linpack flops is what counts in supercomputer ranking. Stuck at .44 Exoflops in 2021.