4 ms·
So my pet theory for this space is that a pure exponent format could be interesting. Like a number would be s * 2 ^ (e / C), where s is a sign bit, and e is a
by im3w1l 1y ago
So my pet theory for this space is that a pure exponent format could be interesting.
Like a number would be s * 2 ^ (e / C), where s is a sign bit, and e is a signed int and C is some constant maybe 2^20 ish for a 32 bit format. Special cases would be carved out for +-0, +-inf and NaN.
- adgjlsfhk1 1y agohttps://en.wikipedia.org/wiki/Logarithmic_number_system https://en.wikipedia.org/wiki/Logarithmic_number_system this has been tried before. The hard part is getting addition to work fast enough.
- im3w1l 1y agoWould addition be slower than for normal floats?
- adgjlsfhk1 1y agomuch. To add 2 lns numbers, you need to essentially compute ln(1+b). There isn't a good algorithm to do so (except for lookup tables for low precision)
- immibis 1y agoSilly idea: what if we ran ML models on pure logarithms and used some completely terrible approximation for addition (maybe just the max function, or max plus a lookup table based on the difference between the two numbers when it's small)
- adgjlsfhk1 1y agoonce you're down to 8 bits or so you can totally do this with a lookup tables (for 8 bits, you need 512 bytes of table which isn't too bad.
- fc417fc802 1y agoIt's been done. [0][1] But the fact that BitNet [2] is possible calls into question the need for it. NNs are surprisingly robust to a wide array of distortions. [3] [0] https://arxiv.org/abs/1603.01025 https://arxiv.org/abs/1603.01025 [1] https://par.nsf.gov/servlets/purl/10163070 https://par.nsf.gov/servlets/purl/10163070 [2] https://arxiv.org/abs/2310.11453 https://arxiv.org/abs/2310.11453 [3] https://arxiv.org/abs/1606.01981 https://arxiv.org/abs/1606.01981
- fc417fc802 1y agoVariants periodically show up on HN. https://news.ycombinator.com/item?id=43181610 https://news.ycombinator.com/item?id=43181610