4 ms·

I was reading Exposing Floating Point today (as Airfoil is on the HN front page and I was perusing the archive of the author). It's a blog explaining the inner

by w-m 3y ago

I was reading Exposing Floating Point today (as Airfoil is on the HN front page and I was perusing the archive of the author). It's a blog explaining the inner workings of floating point representations. About zero values it says [0]: > Yes, the floating point standard specifies both +0.0 and −0.0. This concept is actually useful because it tells us from which “direction” the 0 was approached as a result of storing value too small to be represented in a float. For instance -10e-30f / 10e30f won’t fit in a float, however, it will produce the value of -0.0. The authors of the LLM paper use the values {-1, 0, -1}. Connecting the two ideas, I'm now wondering whether having a 2-bit {-1, -0, 0, 1} representation might have any benefit over the proposed 1.58 bits. Could the additional -0 carry some pseudo-gradient information, ("the 0 leaning towards the negative side")? Also, I've seen 2-bit quantizations being proposed in other LLM quantization papers. What values are they using? [0] https://ciechanow.ski/exposing-floating-point/#zero https://ciechanow.ski/exposing-floating-point/#zero

fabiospampinato 3y ago
I would guess that having 2 zeros is not that useful for NNs, but in general with 2 bits we could encode 4 states, so are there 4 possible states that would be useful to encode? Sure, but would this be better than encoding 3 states? That's the entire question imo. I would guess that 3 states are probably better, because negative/neutral/positive seems the minimal signal that we need these weights to provide.
- eru 3y ago
  You could use a negative-two base, and encode {-2, -1, 0, 1}. See https://en.wikipedia.org/wiki/Negative_base https://en.wikipedia.org/wiki/Negative_base Or you could use the regular positive-two base and encode {-2, -1, 0, 1} the normal way with two's complement.
eru 3y ago
You might also use a basis of negative-two and use two bits to represent {-2, -1, 0, 1}. Negative bases are fun. See https://en.wikipedia.org/wiki/Negative_base https://en.wikipedia.org/wiki/Negative_base
rfoo 3y ago
Interesting, how do you use -0 in the add, then? Is -0+1-1 a 0 or a -0? > Could the additional -0 carry some pseudo-gradient information It looks like training was done on fp32 or bf16. Low-bit quantization is approximated with STE during training. I'd expect training itself cause each point to "polarize" towards 1 or -1. > 2-bit quantizations being proposed Symmetric (i.e. without 0) exponential values were pretty popular IIRC.
- w-m 3y ago
  > how do you use -0 in the add In my mind the two zero values would represent a tiny epsilon around 0, let's say -0.01 and +0.01. Looking at them like this, it would mean +0 +0 -0 = +0 +0 -0 -0 = -0 +1 * +0 = +0 -1 * +0 = -0 Performing addition with the same sign count in each group would be problematic. How to decide on the sign of +0-0 or +1-1, other than flipping a coin?
  - emXdem 3y ago
    [dead]
  - npunt 3y ago
    maybe they could be stored together in two words until they're operated on and lose their pairing?
- paipa 3y ago
  Or use -1, 0, 1/2, 1 where the new half-weight is still a cheap bit shift.
creshal 3y ago
> Could the additional -0 carry some pseudo-gradient information, ("the 0 leaning towards the negative side")? Probably, but is it worth the cost? One of the goals behind BitNet and this paper is to find a way to implement LLMs as efficiently in hardware as possible, and foregoing floating point semantics is a big part of it. I'm not sure if there's a way to encode -0 that doesn't throw out half the performance gains.
- SushiHippie 3y ago
  But if I understand it correctly, they already need to use 2 bits, one for the sign and another one for the value, so there is already one wasted state, which could be used for -0.
  - pennomi 3y ago
    You can pack two trits into three bits, however. So one byte could hold 5 values instead of 4.
    - para_parolu 3y ago
      Can processor perform addition on them effectively?
    - threatripper 3y ago
      How exactly would you do that? 3 states need 1.58 bits which is a tad more than 1.5. Two 3-states have 3²=9 states while three bits only give you 2³=8 states.
      - creshal 3y ago
        I wonder if there's some encoding tricks you can use to reduce it to 8 (or less?) effective states, given that you're only using them with a reduced set of mathematical operations. E.g., can you automatically convert all (-1, 1) to (1, -1) and save one encoded state, since they add up to the same result anyway?
  - 3y ago

by w-m 3y ago

fabiospampinato 3y ago

I would guess that having 2 zeros is not that useful for NNs, but in general with 2 bits we could encode 4 states, so are there 4 possible states that would be useful to encode? Sure, but would this be better than encoding 3 states? That's the entire question imo. I would guess that 3 states are probably better, because negative/neutral/positive seems the minimal signal that we need these weights to provide.

eru 3y ago
You could use a negative-two base, and encode {-2, -1, 0, 1}. See https://en.wikipedia.org/wiki/Negative_base https://en.wikipedia.org/wiki/Negative_base Or you could use the regular positive-two base and encode {-2, -1, 0, 1} the normal way with two's complement.

eru 3y ago

You might also use a basis of negative-two and use two bits to represent {-2, -1, 0, 1}. Negative bases are fun. See https://en.wikipedia.org/wiki/Negative_base https://en.wikipedia.org/wiki/Negative_base

rfoo 3y ago

Interesting, how do you use -0 in the add, then? Is -0+1-1 a 0 or a -0? > Could the additional -0 carry some pseudo-gradient information It looks like training was done on fp32 or bf16. Low-bit quantization is approximated with STE during training. I'd expect training itself cause each point to "polarize" towards 1 or -1. > 2-bit quantizations being proposed Symmetric (i.e. without 0) exponential values were pretty popular IIRC.

w-m 3y ago
> how do you use -0 in the add In my mind the two zero values would represent a tiny epsilon around 0, let's say -0.01 and +0.01. Looking at them like this, it would mean +0 +0 -0 = +0 +0 -0 -0 = -0 +1 * +0 = +0 -1 * +0 = -0 Performing addition with the same sign count in each group would be problematic. How to decide on the sign of +0-0 or +1-1, other than flipping a coin?
- emXdem 3y ago
  [dead]
- npunt 3y ago
  maybe they could be stored together in two words until they're operated on and lose their pairing?
paipa 3y ago
Or use -1, 0, 1/2, 1 where the new half-weight is still a cheap bit shift.

creshal 3y ago

> Could the additional -0 carry some pseudo-gradient information, ("the 0 leaning towards the negative side")? Probably, but is it worth the cost? One of the goals behind BitNet and this paper is to find a way to implement LLMs as efficiently in hardware as possible, and foregoing floating point semantics is a big part of it. I'm not sure if there's a way to encode -0 that doesn't throw out half the performance gains.

SushiHippie 3y ago
But if I understand it correctly, they already need to use 2 bits, one for the sign and another one for the value, so there is already one wasted state, which could be used for -0.
- pennomi 3y ago
  You can pack two trits into three bits, however. So one byte could hold 5 values instead of 4.
  - para_parolu 3y ago
    Can processor perform addition on them effectively?
  - threatripper 3y ago
    How exactly would you do that? 3 states need 1.58 bits which is a tad more than 1.5. Two 3-states have 3²=9 states while three bits only give you 2³=8 states.
    - creshal 3y ago
      I wonder if there's some encoding tricks you can use to reduce it to 8 (or less?) effective states, given that you're only using them with a reduced set of mathematical operations. E.g., can you automatically convert all (-1, 1) to (1, -1) and save one encoded state, since they add up to the same result anyway?
- 3y ago