3 ms·
> Could the additional -0 carry some pseudo-gradient information, ("the 0 leaning towards the negative side")? Probably, but is it worth the cost? One of the g
by creshal 3y ago
> Could the additional -0 carry some pseudo-gradient information, ("the 0 leaning towards the negative side")?
Probably, but is it worth the cost? One of the goals behind BitNet and this paper is to find a way to implement LLMs as efficiently in hardware as possible, and foregoing floating point semantics is a big part of it. I'm not sure if there's a way to encode -0 that doesn't throw out half the performance gains.
- SushiHippie 3y agoBut if I understand it correctly, they already need to use 2 bits, one for the sign and another one for the value, so there is already one wasted state, which could be used for -0.
- pennomi 3y agoYou can pack two trits into three bits, however. So one byte could hold 5 values instead of 4.
- para_parolu 3y agoCan processor perform addition on them effectively?
- threatripper 3y agoHow exactly would you do that? 3 states need 1.58 bits which is a tad more than 1.5. Two 3-states have 3²=9 states while three bits only give you 2³=8 states.
- creshal 3y agoI wonder if there's some encoding tricks you can use to reduce it to 8 (or less?) effective states, given that you're only using them with a reduced set of mathematical operations. E.g., can you automatically convert all (-1, 1) to (1, -1) and save one encoded state, since they add up to the same result anyway?
- Manabu-eo 3y agoYou can use a bit for zero or non-zero and then use bits only for providing the sign to non-zero values, for example. The sign part will be variable length but can probably be made very fast with hardware support.