4 ms·
Interesting, how do you use -0 in the add, then? Is -0+1-1 a 0 or a -0? > Could the additional -0 carry some pseudo-gradient information It looks like trainin
by rfoo 3y ago
Interesting, how do you use -0 in the add, then? Is -0+1-1 a 0 or a -0?
> Could the additional -0 carry some pseudo-gradient information
It looks like training was done on fp32 or bf16. Low-bit quantization is approximated with STE during training. I'd expect training itself cause each point to "polarize" towards 1 or -1.
> 2-bit quantizations being proposed
Symmetric (i.e. without 0) exponential values were pretty popular IIRC.
- w-m 3y ago> how do you use -0 in the add In my mind the two zero values would represent a tiny epsilon around 0, let's say -0.01 and +0.01. Looking at them like this, it would mean +0 +0 -0 = +0 +0 -0 -0 = -0 +1 * +0 = +0 -1 * +0 = -0 Performing addition with the same sign count in each group would be problematic. How to decide on the sign of +0-0 or +1-1, other than flipping a coin?
- paipa 3y agoOr use -1, 0, 1/2, 1 where the new half-weight is still a cheap bit shift.