3 ms·
The issue is when you add a very small number to a larger number. This occurs in neural networks when accumulating gradients during learning. Many of the gradie
by Houshalter 9y ago
The issue is when you add a very small number to a larger number. This occurs in neural networks when accumulating gradients during learning. Many of the gradients are very small, and when adding them repeatedly they get rounded down. There was a paper studying the effect, and I think it was after about 14 bits of precision the accuracy is very diminished. But with stochastic rounding they could get down to very few bits.
- scottlegrand2 9y agoAll of this is fantastic if you are a deep learning Jedi Master. But there's a very small supply of those sorts. I really worry about all these reduced floating-point representations when they are made use of by people who mostly understand deep learning through tinkering with existing tensorflow tutorials. FP32 seems like a relative sweet spot with sufficient dynamic range to let most amateurs avoid getting trapped in the weeds. I could probably be persuaded to believe that FP24 is sufficient as well. But I suspect that once you get down to 16 bits or so you have to do all sorts of stochastic rounding / dithering that is beyond the skill set of most data scientists. And that's because I suspect many of them don't really get dynamic range. Which then leads me to believe that what we really need here is the equivalent of R for machine learning.