4 ms·

Anecdotally, it seems most models can be quantized to 8 bits without much loss of accuracy, and fixed point arithmetic requires much less hardware. Training is

Anecdotally, it seems most models can be quantized to 8 bits without much loss of accuracy, and fixed point arithmetic requires much less hardware. Training is still done with floating point though.

ChuckMcM 10y ago
This, when you get right down to it a lot of models do fine with only 256 unique weights.
- gcr 10y ago
  Yeah...you could implement 16-bit multiplication/addition as two 8-bit multiplications plus carry... so worst case, if you want 16-bit multiplies, you implement it yourself
  - stephencanon 10y ago
    Four multiplications, not two (three if you only need the low 16b of the result).
    - yorwba 10y ago
      You can get the full 32-bit result in only three multiplications using https://en.wikipedia.org/wiki/Karatsuba_algorithm https://en.wikipedia.org/wiki/Karatsuba_algorithm , but it requires more additions.
      - stephencanon 10y ago
        Yeah, in practice Karatsuba isn't a performant option for small operands unless your multiplier is catastrophically slow. (And it still doesn't get you to two multiplications.)
- alttab 10y ago
  Agreed - however as we progress I expect a comment like this to be akin to Bill Gate's 64K comment.
  - ChuckMcM 10y ago
    Almost certainly, and depths would have to increase. Like all series expansions the coefficients on later terms have less and less impact so their dynamic range is less and less important to the final value. But the dynamic range on the initial terms is proportionately important. I expect the dynamic range of the weights will turn out to be logarithmic with respect to overall depth of the network.
  - mannigfaltig 10y ago
    The brain appears to spend about 4.7 bits per synapse (26 discernible states, given the noisy computation environment of the brain); so it seems to be plenty enough for general intelligence. This could, of course, merely be a biological limit and on silicon more fine-grained weights might be the optimum. Here is another paper demonstrating very good results with just 6 bit gradients: https://arxiv.org/abs/1606.06160 https://arxiv.org/abs/1606.06160
pklausler 10y ago
Indeed, any 8-bit x 8-bit function with an 8-bit result is just a 64KiB look-up table.
koja86 10y ago
There are hints that even less than 8 bits per weight might be usable (for certain cases and on custom hw). Not sure if it's practical but it is definitely interesting. https://arxiv.org/pdf/1502.02551.pdf https://arxiv.org/pdf/1502.02551.pdf https://arxiv.org/pdf/1610.00324.pdf https://arxiv.org/pdf/1610.00324.pdf I wanted to have some basic idea about hardware so I did some "research" (googling) and ended up giving a short informal talk. My slides with some links are here: http://ml.kjx.cz/hw4nn2017.pdf http://ml.kjx.cz/hw4nn2017.pdf

Anecdotally, it seems most models can be quantized to 8 bits without much loss of accuracy, and fixed point arithmetic requires much less hardware. Training is still done with floating point though.

ChuckMcM 10y ago

This, when you get right down to it a lot of models do fine with only 256 unique weights.

gcr 10y ago
Yeah...you could implement 16-bit multiplication/addition as two 8-bit multiplications plus carry... so worst case, if you want 16-bit multiplies, you implement it yourself
- stephencanon 10y ago
  Four multiplications, not two (three if you only need the low 16b of the result).
  - yorwba 10y ago
    You can get the full 32-bit result in only three multiplications using https://en.wikipedia.org/wiki/Karatsuba_algorithm https://en.wikipedia.org/wiki/Karatsuba_algorithm , but it requires more additions.
    - stephencanon 10y ago
      Yeah, in practice Karatsuba isn't a performant option for small operands unless your multiplier is catastrophically slow. (And it still doesn't get you to two multiplications.)
alttab 10y ago
Agreed - however as we progress I expect a comment like this to be akin to Bill Gate's 64K comment.
- ChuckMcM 10y ago
  Almost certainly, and depths would have to increase. Like all series expansions the coefficients on later terms have less and less impact so their dynamic range is less and less important to the final value. But the dynamic range on the initial terms is proportionately important. I expect the dynamic range of the weights will turn out to be logarithmic with respect to overall depth of the network.
- mannigfaltig 10y ago
  The brain appears to spend about 4.7 bits per synapse (26 discernible states, given the noisy computation environment of the brain); so it seems to be plenty enough for general intelligence. This could, of course, merely be a biological limit and on silicon more fine-grained weights might be the optimum. Here is another paper demonstrating very good results with just 6 bit gradients: https://arxiv.org/abs/1606.06160 https://arxiv.org/abs/1606.06160

pklausler 10y ago

Indeed, any 8-bit x 8-bit function with an 8-bit result is just a 64KiB look-up table.

koja86 10y ago

There are hints that even less than 8 bits per weight might be usable (for certain cases and on custom hw). Not sure if it's practical but it is definitely interesting. https://arxiv.org/pdf/1502.02551.pdf https://arxiv.org/pdf/1502.02551.pdf https://arxiv.org/pdf/1610.00324.pdf https://arxiv.org/pdf/1610.00324.pdf I wanted to have some basic idea about hardware so I did some "research" (googling) and ended up giving a short informal talk. My slides with some links are here: http://ml.kjx.cz/hw4nn2017.pdf http://ml.kjx.cz/hw4nn2017.pdf