3 ms·
Interesting comment on machine learning. My understanding is that they don't train with half precision, but quantisation allows even integer values to be used
by psychphysic 4y ago
Interesting comment on machine learning.
My understanding is that they don't train with half precision, but quantisation allows even integer values to be used for inference?
Yet the reasoning still makes sense. Can training take place at half precision or even if you intend to run inference at halfp do you want doublep for training?
- zone411 4y agoTraining with half precision (or mixed precision) is common.
- mFixman 4y agoMy guess is that very small and very large values in the weights are already trained away due to the regularisation of the cost function, so insignificant changes in a network don't tend to have significant changes in the output. You gain more by being able to run gradient descent faster than by having higher-precision floats.
- Nevermark 4y agoAbsolutely. And beyond weight regularization, for any weighting followed by a sigmoid or other squashing function, large weights simply tend to saturate the squashing function and there is very little gradient (quickly effectively zero) to benefit from increasing the weight value past that point.