2 ms·
Silly idea: what if we ran ML models on pure logarithms and used some completely terrible approximation for addition (maybe just the max function, or max plus a
by immibis 1y ago
Silly idea: what if we ran ML models on pure logarithms and used some completely terrible approximation for addition (maybe just the max function, or max plus a lookup table based on the difference between the two numbers when it's small)
- adgjlsfhk1 1y agoonce you're down to 8 bits or so you can totally do this with a lookup tables (for 8 bits, you need 512 bytes of table which isn't too bad.
- fc417fc802 1y agoIt's been done. [0][1] But the fact that BitNet [2] is possible calls into question the need for it. NNs are surprisingly robust to a wide array of distortions. [3] [0] https://arxiv.org/abs/1603.01025 https://arxiv.org/abs/1603.01025 [1] https://par.nsf.gov/servlets/purl/10163070 https://par.nsf.gov/servlets/purl/10163070 [2] https://arxiv.org/abs/2310.11453 https://arxiv.org/abs/2310.11453 [3] https://arxiv.org/abs/1606.01981 https://arxiv.org/abs/1606.01981