3 ms·
Got any sources? Was thinking about buying one just for the tensor cores, but if this is the case I probably won't.
by jongomez 8y ago
Got any sources? Was thinking about buying one just for the tensor cores, but if this is the case I probably won't.
- bitL 8y agoYou can even see it in author's comments in the original article: "When I first looked at fp16 Inception3 was the largest model I could train. Inception4 blew up until I went back to fp32. Mixed precision needs extra care, scaling of gradients and such. Still I think it is a good thing. What I really want to test is model size reduction for inference with TensorRT targeted to tensorcores. I think that is probably the best use case. Non-linear optimization is just too susceptible to precision loss." There was also some NVidia video presentation recommending mixed FP32/FP16 training instead of pure FP16.
- option 8y agoMixed precision training can give you tensor core speedups. Paper: https://arxiv.org/abs/1710.03740 https://arxiv.org/abs/1710.03740 Toolkit which implements it on top of Tensorflow: https://github.com/NVIDIA/OpenSeq2Seq https://github.com/NVIDIA/OpenSeq2Seq