3 ms·
It's worse than that, the FP16 MAD in GTX 1080 TI is significantly slower than converting 2 FP16 numbers to FP32, performing FP32 math on them, and accumulating
by scottlegrand2 9y ago
It's worse than that, the FP16 MAD in GTX 1080 TI is significantly slower than converting 2 FP16 numbers to FP32, performing FP32 math on them, and accumulating results in FP32. Had it been the same speed, I don't think I would be anywhere near as annoyed as I am with this crippling.
That said, at Vega's 26 or so FP16 TFLOPS, it wouldn't be hard for NVIDIA to release a GeForce Volta with 30-40 tensor core TFLOPS, that both stomped on Vega and remained significantly inferior to V100. Given how hard it is to program Volta optimally, I'm surprised they haven't done so already, if only as a Titan XV Edition that can only be purchased from their website.
- dharma1 9y agoYeah, will be interesting to see if they can bring the price of GPUs with tensor cores down, from what I've read it's too expensive to make them right now for consumer market. BTW - do you happen to know what DL frameworks currently support mixed precision training with Volta tensor cores? Curious to see if AWS V100 instances can really do 120 TFLOPS as advertised. I think latest versions of CUDA/CuDNN support Volta now?