4 ms·
Probably even 4 bit values suffice. Invent some binary operations that exhibit some smoothness or continuousness such that gradient descent still works, because
by _nalply 3y ago
Probably even 4 bit values suffice. Invent some binary operations that exhibit some smoothness or continuousness such that gradient descent still works, because mostly a small change should have only a small effect. I think mathematicians would say functions that are differentiable.
- tayo42 3y agoQuantization is done with 4 bits, unless you meant something else?
- _nalply 3y agoPerhaps quantization with 4 bits is a special case of what I meant. With 4 bits you have 16 different weight values but these values must not be quantized. They could be anything provided gradient descent is possible. Have some combination of binary operations instead of mathematical operations. What these operations do exactly, does not really matter, as long as gradient descent/backpropagation stays possible. In other words, the model will find a way to use the weights correctly.