3 ms·
Their best performing 4-bit number format uses 1 sign bit, 3 exponent bits, and no mantissa bits! Ie. All weights, activations and gradients become powers of t
by londons_explore 3y ago
Their best performing 4-bit number format uses 1 sign bit, 3 exponent bits, and no mantissa bits!
Ie. All weights, activations and gradients become powers of two! Which means all multiplications become simple bit shifts. That really changes mathematics and silicon design.
- Dylan16807 3y agoDoes it really make much of a difference? You're usually feeding a ton of multiplies into an accumulator. You can handle one or two mantissa bits as the same bit shifting except that it outputs two or three numbers to accumulate. And accumulators are very easy to scale. Also in the extreme I've seen powers of 4 get used.