3 ms·
I'm a computational scientist. Do ML problems not deal with problems that are sensitive input precision? Is it too naive to say if they don't, does one really n
by noobermin 8y ago
I'm a computational scientist. Do ML problems not deal with problems that are sensitive input precision? Is it too naive to say if they don't, does one really need ML for said problem over just plain old fitting and stats?
- Tuna-Fish 8y agoThese are not used for data but for the computation of the internal coefficients. Said coefficients would be stored as F32:s, and the job of modifying them involves computing a lot of multiplications, none of which need to be that precise.
- noobermin 8y agoWouldn't conversion between the steps lead to round off?
- Tuna-Fish 8y agoConversion from Bfloat16 to f32 is just extending the mantissa with zeroes. Basically, what they are computing is: f32 acc = C_a0 * C_b0 + Ca1 * Cb1 + Ca2 * Cb2 ... with very many coefficients, all of which are Bfloat16. The precision of the coefficients is not that important, but they can be of substantially different magnitude, so the coefficients can use few bits in the mantissa but the accumulator needs to be wider.