3 ms·
There are some crazy 1/1.5 bit quants now. If you're curious I'll try to dig up the papers I was reading. 1.5bit can be done to existing models. The 1 bit (and
by freeqaz 2y ago
There are some crazy 1/1.5 bit quants now. If you're curious I'll try to dig up the papers I was reading.
1.5bit can be done to existing models. The 1 bit (and less than 1 bit iirc) requires training a model from scratch.
Still, the idea that we can have giant models running in tiny amounts of RAM is not completely far fetched at this point.
- gpm 2y agoYeah, I'm broadly aware and have seen a few of the papers, though I definitely don't try and track the state of the art here closely. My impression and experience trying low bit quants (which could easily be outdated by now) is that you are/were better off with a smaller model and a less aggressive quantization (provided you have access to said smaller model with otherwise equally good training). If that's changed I'd be interested to hear about it, but definitely don't want to make work for you digging up papers.