4 ms·
So Google still won't reveal what kind of floating-point TPU 2 actually runs. That is really really really strange. I cannot help but suspect it's some sort of
by scottlegrand2 9y ago
So Google still won't reveal what kind of floating-point TPU 2 actually runs. That is really really really strange. I cannot help but suspect it's some sort of non-standard floating point but that's easy to clear up, no?
- deepnotderp 9y agoPretty sure it's a mini float of some sort.
- monocasa 9y agoBack in the day it was fairly common to not be compliant with IEEE-754 to cut down gate count. Like just giving the wrong results on de-normals, fixed to one rounding mode, etc. I bet they're doing something like that.
- Houshalter 9y agoStochastic rounding is also important for doing these algorithms in very low precision. Without stochastic rounding, even very small rounding errors accumulate enough to destroy the accuracy.
- deepnotderp 9y agoStochastic rounding shouldn't be entirely necessary for floating point though, no?
- Houshalter 9y agoHow does floating point change anything? You still only have so many bits of precision, and the lower bits need to be rounded. In fact floating point has a lot less precision, since many of the bits are needed to store the exponent.
- moonbug22 9y agoRange.
- deepnotderp 9y agoThe range of the exponent. Stochastic rounding is intended to prevent bias towards zeroes which paralyze learning, but with floating point this is less of an issue since you only round the mantissa.
- Houshalter 9y agoThe issue is when you add a very small number to a larger number. This occurs in neural networks when accumulating gradients during learning. Many of the gradients are very small, and when adding them repeatedly they get rounded down. There was a paper studying the effect, and I think it was after about 14 bits of precision the accuracy is very diminished. But with stochastic rounding they could get down to very few bits.
- scottlegrand2 9y agoAll of this is fantastic if you are a deep learning Jedi Master. But there's a very small supply of those sorts. I really worry about all these reduced floating-point representations when they are made use of by people who mostly understand deep learning through tinkering with existing tensorflow tutorials. FP32 seems like a relative sweet spot with sufficient dynamic range to let most amateurs avoid getting trapped in the weeds. I could probably be persuaded to believe that FP24 is sufficient as well. But I suspect that once you get down to 16 bits or so you have to do all sorts of stochastic rounding / dithering that is beyond the skill set of most data scientists. And that's because I suspect many of them don't really get dynamic range. Which then leads me to believe that what we really need here is the equivalent of R for machine learning.
- puzzle 9y agoMaybe a paper is forthcoming? At least Jeff Dean has confirmed that it's some kind of FP, not e.g. decimal arithmetic. :-)
- peter303 9y agoI'd use one of the debugged/tested open-source designs. Its so easy to make a mistake with complicated specifications of IEEE standards.
- peter303 9y agoI'd use one of the debugged/tested open-source designs. Its so easy to make a mistake with complicated specifications of IEEE standards.