5 ms·
"The TPU only operates on 8-bit integers" The 8 bit part is fine, but integers? what the hell. that's a new one for me
by make3 10y ago
"The TPU only operates on 8-bit integers" The 8 bit part is fine, but integers?
what the hell. that's a new one for me
- mtgx 10y agoNvidia has shipped a couple of INT8 inference cards as well: http://www.anandtech.com/show/10675/nvidia-announces-tesla-p40-tesla-p4 http://www.anandtech.com/show/10675/nvidia-announces-tesla-p...
- hamilyon2 10y agoYes, how do they perform? Linked article cites 47 TOPS on 8 bit integer arithmetic and they are useful for training too.
- struct 10y agoI imagine that once they've trained the floating-point models, they'll then quantize them into integers to make inference faster. It's not something I've done, but I imagine that the limited range of the of integers may cause problems (though they say in the paper that the 16-bit product can be accumulated to something that's 32-bit). The features to do this will be coming fairly soon to regular TensorFlow too.[1] [1] https://youtu.be/0r9w3V923rk?list=PLOU2XLYxmsIKGc_NBoIhTn2Qhraji53cv&t=1617 https://youtu.be/0r9w3V923rk?list=PLOU2XLYxmsIKGc_NBoIhTn2Qh...
- dlubarov 10y agoAnecdotally, it seems most models can be quantized to 8 bits without much loss of accuracy, and fixed point arithmetic requires much less hardware. Training is still done with floating point though.
- ChuckMcM 10y agoThis, when you get right down to it a lot of models do fine with only 256 unique weights.
- gcr 10y agoYeah...you could implement 16-bit multiplication/addition as two 8-bit multiplications plus carry... so worst case, if you want 16-bit multiplies, you implement it yourself
- stephencanon 10y agoFour multiplications, not two (three if you only need the low 16b of the result).
- yorwba 10y agoYou can get the full 32-bit result in only three multiplications using https://en.wikipedia.org/wiki/Karatsuba_algorithm https://en.wikipedia.org/wiki/Karatsuba_algorithm , but it requires more additions.
- stephencanon 10y agoYeah, in practice Karatsuba isn't a performant option for small operands unless your multiplier is catastrophically slow. (And it still doesn't get you to two multiplications.)
- alttab 10y agoAgreed - however as we progress I expect a comment like this to be akin to Bill Gate's 64K comment.
- ChuckMcM 10y agoAlmost certainly, and depths would have to increase. Like all series expansions the coefficients on later terms have less and less impact so their dynamic range is less and less important to the final value. But the dynamic range on the initial terms is proportionately important. I expect the dynamic range of the weights will turn out to be logarithmic with respect to overall depth of the network.
- 10y ago
- pklausler 10y agoIndeed, any 8-bit x 8-bit function with an 8-bit result is just a 64KiB look-up table.
- koja86 10y agoThere are hints that even less than 8 bits per weight might be usable (for certain cases and on custom hw). Not sure if it's practical but it is definitely interesting. https://arxiv.org/pdf/1502.02551.pdf https://arxiv.org/pdf/1502.02551.pdf https://arxiv.org/pdf/1610.00324.pdf https://arxiv.org/pdf/1610.00324.pdf I wanted to have some basic idea about hardware so I did some "research" (googling) and ended up giving a short informal talk. My slides with some links are here: http://ml.kjx.cz/hw4nn2017.pdf http://ml.kjx.cz/hw4nn2017.pdf
- woodson 10y agoTake a look at Google's gemmlowp low-precision GEMM library: https://github.com/google/gemmlowp https://github.com/google/gemmlowp (Used in tensorflow)
- throwaway71958 10y agoNote however, that on Intel it's actually slower than run off the mill float32 linear algebra library like Eigen or OpenBLAS. Its main forte seems to be ARM.