4 ms·
TF32 = FP32 range + FP16 precision
by p1esk 5y ago
TF32 = FP32 range + FP16 precision
- cjbgkagh 5y agoWhy not call it TF19 then.
- p1esk 5y agoBecause your existing FP32 models should run fine when converted to TF32, so TF32 is equivalent to FP32 as far as DL practitioners are concerned.
- cjbgkagh 5y agoThere is a lot of redundancy in DL that forgives all manner of sins; think it’s sneaky.
- 37ef_ced3 5y agoBecause it's 32-bits wide in memory. The effective mantissa is like FP16 but it's padded out to be the same size as FP32. In other words, there's 1 sign bit, 8 exponent bits, 10 mantissa bits that are USED, and 13 mantissa bits that are IGNORED. 1 + 8 + 10 + 13 = 32 The 13 ignored mantissa bits are part of the memory image: they pad the number out to 32-bit alignment.
- cjbgkagh 5y agoBut the user never sees that memory right? Doesn't it go in FP32 and come out FP32? I still think it's deceptive marketing.
- bcatanzaro 5y agoThe user does see 32-bits and all bits are used because all the additions (and other operations besides the multiply in matrix ops) are in FP32. So the bottom bits are populated with useful information.