3 ms·

At this rate, we're going to end up with FP1 (1-bit floating point) numbers... I guess that's nonsensical. 1-bit in all FP is the sign bit. So I guess the mini

At this rate, we're going to end up with FP1 (1-bit floating point) numbers... I guess that's nonsensical. 1-bit in all FP is the sign bit. So I guess the minimum size is 2-bit FP (1-bit sign + 1-bit exponent + 0-bit implicit 1 mantissa)

kortex 4y ago
At FP2, you are probably better off with {-1, 0, 1, NaN} (sign+mantissa) rather than sign/exponent. You basically bit pack. FP3 gives you sign, 1x "exponent", 1x mantissa, so still kinda bit packing. I could see FP4 with sign, 1x exponent, 2x mantissa. Exponent would really just be a 4x multiplier, giving +/-0,1,2,3,4,8,12 Or invert all those, so you are expressing common fractions on 0..1
- dragontamer 4y ago
  Real life has the E3 series: 1, 2.2, 4.7, and then 10, 22, 47, 100, 220, 470, 1000, etc. Etc. EEs would recognize these values to be the Preferred Resistors values for projects (though more commonly the E6 series is used in projects, the E3 and E1 values are preferred) That's 3 values per decade, which is slightly more dispersed than a FP4 that consists of 1 sign + 3 exponent + 0 (implicit mantissa 1 bit). Or the values -128, -64, -32, -16, -8, -4, -2, -1, 1, 2, ... 128. Maybe we can take -128 and call that zero instead, cause zero is useful. -------- Given how even E3 is still useful in real world electrical engineering problems, I'm more inclined to allocate more bits to the exponent than the mantissa.
  - ben_w 4y ago
    > Real life has the E3 series: 1, 2.2, 4.7, and then 10, 22, 47, 100, 220, 470, 1000, etc. Etc. Took me until today to realise that sequence is a rounded version of 10^(n/3) for integer n.
    - simcop2387 4y ago
      Almost so, there are some manual adjustments to help with overlap from tolerances at various places, but that pure math layout of it would be where you'd probably want to do it for ml
  - stkdump 4y ago
    E3 is only relevant to decimal numbers. I don't see how computing can benefit from it, unless base conversion for human interaction is a particularly large part of the problem. And there is a reason why stuff like BCD isn't used anywhere near anything resembling high performance computing: it's practically never worth it.
- Dylan16807 4y ago
  If you're going to bother doing floats you should probably make them balanced around 1. And exponent seems to be much more important for these small sizes. The first paper that shows up for FP4 almost has negative mantissa bits. Their encoding has 0, 1/64, 1/16, 1/4, 1, 4, 16, 64.
SideQuark 4y ago
1 bit would work fine - make the values represent +-1 or so
varispeed 4y ago
That once we get into asymmetrical number coding so that you could use numbers that take fraction of bits.
visarga 4y ago
I think I read somewhere it only goes as low as int4. Can't find the reference.
dimatura 4y ago
Binary neural networks, where weights and/or activations are just 0/1s, are an active research area. In theory they could be implemented very efficiently in hardware. But in contrast to FP16 (or to some extent, int8), just quantizing FP32 to 1 bit doesn't work very well. There have been successful methods in practice. There was a company called Xnor.ai that was built partially around this technology, but it was sold to Apple a couple years ago. I don't know what's the current SOTA in this area, though.
- nutanc 4y ago
  We have created a binary embedding to help us with the embedding sizes. I think in the future we will see a lot more research in reducing the model and embedding sizes. https://medium.com/ozonetel-ai/compressing-bert-sentence-embeddings-6120c84f5f4c https://medium.com/ozonetel-ai/compressing-bert-sentence-emb...
- dyno12345 4y ago
  *binarized
mike_hock 4y ago
So +0, +infinity, -0, -infinity, no NaNs?
NikkiA 4y ago
1-bit unsigned float would probably be useful to some

kortex 4y ago

At FP2, you are probably better off with {-1, 0, 1, NaN} (sign+mantissa) rather than sign/exponent. You basically bit pack. FP3 gives you sign, 1x "exponent", 1x mantissa, so still kinda bit packing. I could see FP4 with sign, 1x exponent, 2x mantissa. Exponent would really just be a 4x multiplier, giving +/-0,1,2,3,4,8,12 Or invert all those, so you are expressing common fractions on 0..1

dragontamer 4y ago
Real life has the E3 series: 1, 2.2, 4.7, and then 10, 22, 47, 100, 220, 470, 1000, etc. Etc. EEs would recognize these values to be the Preferred Resistors values for projects (though more commonly the E6 series is used in projects, the E3 and E1 values are preferred) That's 3 values per decade, which is slightly more dispersed than a FP4 that consists of 1 sign + 3 exponent + 0 (implicit mantissa 1 bit). Or the values -128, -64, -32, -16, -8, -4, -2, -1, 1, 2, ... 128. Maybe we can take -128 and call that zero instead, cause zero is useful. -------- Given how even E3 is still useful in real world electrical engineering problems, I'm more inclined to allocate more bits to the exponent than the mantissa.
- ben_w 4y ago
  > Real life has the E3 series: 1, 2.2, 4.7, and then 10, 22, 47, 100, 220, 470, 1000, etc. Etc. Took me until today to realise that sequence is a rounded version of 10^(n/3) for integer n.
  - simcop2387 4y ago
    Almost so, there are some manual adjustments to help with overlap from tolerances at various places, but that pure math layout of it would be where you'd probably want to do it for ml
- stkdump 4y ago
  E3 is only relevant to decimal numbers. I don't see how computing can benefit from it, unless base conversion for human interaction is a particularly large part of the problem. And there is a reason why stuff like BCD isn't used anywhere near anything resembling high performance computing: it's practically never worth it.
Dylan16807 4y ago
If you're going to bother doing floats you should probably make them balanced around 1. And exponent seems to be much more important for these small sizes. The first paper that shows up for FP4 almost has negative mantissa bits. Their encoding has 0, 1/64, 1/16, 1/4, 1, 4, 16, 64.

SideQuark 4y ago

1 bit would work fine - make the values represent +-1 or so

varispeed 4y ago

That once we get into asymmetrical number coding so that you could use numbers that take fraction of bits.

visarga 4y ago

I think I read somewhere it only goes as low as int4. Can't find the reference.

dimatura 4y ago

Binary neural networks, where weights and/or activations are just 0/1s, are an active research area. In theory they could be implemented very efficiently in hardware. But in contrast to FP16 (or to some extent, int8), just quantizing FP32 to 1 bit doesn't work very well. There have been successful methods in practice. There was a company called Xnor.ai that was built partially around this technology, but it was sold to Apple a couple years ago. I don't know what's the current SOTA in this area, though.

nutanc 4y ago
We have created a binary embedding to help us with the embedding sizes. I think in the future we will see a lot more research in reducing the model and embedding sizes. https://medium.com/ozonetel-ai/compressing-bert-sentence-embeddings-6120c84f5f4c https://medium.com/ozonetel-ai/compressing-bert-sentence-emb...
dyno12345 4y ago
*binarized

mike_hock 4y ago

So +0, +infinity, -0, -infinity, no NaNs?

NikkiA 4y ago

1-bit unsigned float would probably be useful to some