3 ms·
They keep calling them 1-bit LLMs, but they're really 1-trit LLMs. If you can have 3 states, it's not a BInary uniT, it's a TRInary uniT. I don't think that th
by bunderbunder 2y ago
They keep calling them 1-bit LLMs, but they're really 1-trit LLMs. If you can have 3 states, it's not a BInary uniT, it's a TRInary uniT.
I don't think that this is just a nit, it implies a real mismatch between how these models work, and how modern computing hardware works. People have built ternary computers in the past, but to the best of my knowledge nobody's made a serious attempt at it in half a century.
You can always use two bits to store a ternary value, of course. But then it wouldn't be a 1-bit LLM, now, would it? And that doubling in compute resources required would make a tangible difference in how one has to think about potential efficiency improvements. Also, ternary tritwise logic implemented on binary hardware is unlikely to be anywhere near as efficient as binary bitwise logic on hardware that was specifically engineered for the purpose. This leaves me thinking that the research teams involved continuing to refer to these as 1-bit LLMs must be interpreted as knowingly engaging in dishonest marketing tactics.
- freeone3000 2y agoIt’s stored in and calculated using a floating-point value, and uses log(2) bits. So it’s ~1.2 bits with float packing over 32m parameters.
- IanCal 2y agoI don't think it's quite so clear cut, and the papers and names are often a bit more precise anyway. BitNet is (I think) actually one bit per parameter. BitNet 1.58b isn't, but then to be fair isn't describing itself as a 1 bit llm. I'm less sure but it seems OneBit is 1 bit. One of the processes mentioned in the article is a mix of one and two bits for the weights.
- orlp 2y agoPerhaps in the future we will see MsoTriState neural networks: https://learn.microsoft.com/en-us/dotnet/api/microsoft.office.core.msotristate https://learn.microsoft.com/en-us/dotnet/api/microsoft.offic... For those unaware, the MsoTriState is fairly self-explanatory: it is a tri-state boolean type with five possible values.
- neolefty 2y agoThe article describes both. The 1,0,-1 networks are "1.58 bits" — that is, log2(3). But yeah it mostly focuses on 1-bit networks.
- corysama 2y agoNot long ago I managed to get someone working on sub-1-bit models to come out of the woodwork. https://news.ycombinator.com/item?id=39865855 https://news.ycombinator.com/item?id=39865855