5 ms·
I wonder whether quantization to 1-bit would be functional?
by skykooler 3y ago
I wonder whether quantization to 1-bit would be functional?
- deleted 3y ago[deleted]
- gumby 3y agoThere was a paper from the Allen institute from around 2017 successfully using 1 bit quantization but I can’t find it right now. We started using it where I was working at the time but I’m no longer there so I don’t know how it all turned out.
- orbital-decay 3y agoFunctional? Sure, it will work. Useful? Unlikely. It would be surprising to see quantization of pretrained models to surpass the existing research around binary NNs (trained from scratch as 1-bit while using full precision for the most critical parts)
- rolisz 3y agoIf we get 1-bit quantization, wouldn't it be basically a bunch of nested if's and else's?
- mst_moonshine 3y agoThose networks are named as discrete neural networks. There are already research on those, mainly for (homomorphic) encryption purposes (because it’s much easier to homomorphically encrypt NNs of just 0’s and 1’s than normal NNs).
- patrakov 3y agoI predict that even less-than-1-bit viable quantization methods will be found eventually. Of course, the bits-per-weight figure should be interpreted as something average: e.g., group-quantizing 32 weights into 24 bits would be 0.75 bits per weight.
- janwas 3y agoYes :) https://arxiv.org/abs/2310.16795 https://arxiv.org/abs/2310.16795