4 ms·
The performance loss from BiLLM is disastrous. It's basically useless. No one would ever want to use the resulting models. They hide their main results in the a
by light_hue_1 2y ago
The performance loss from BiLLM is disastrous. It's basically useless. No one would ever want to use the resulting models. They hide their main results in the appendix: Table 8. page 15. https://arxiv.org/pdf/2402.04291 https://arxiv.org/pdf/2402.04291
I won't go over the entire table in detail, but PIQA, BoolQ, HellaSwag, and WinoGrande should in the mid-to-high 70s for LLaMa2-7B. They drop that to 58, 62, 32, and 51. There are 700M parameter models that perform much better.
What they should have reported is effective number of parameters. Does LLaMa2-7B with their quantization method outperform a model that has amount of computation but uses that compute with say.. 16-bit quantization? If the answer is no, and it seems like it very clearly is, then the method is wholly worthless. Just use a smaller model to begin with.
The BitNet paper is better. But for some reason they only consider very small models. It's the obvious question and in their FAQ they don't provide a solid answer to it. Despite having all of the compute resources of MS. They could have easily run this experiment in the past year; I'm suspicious.
- imtringued 2y agoBiLLM is about post training quantization and BitNet trains models from scratch. You do realize that one of those is going to give significantly worse results and the other is going to be significantly more expensive, well into the millions of dollars?
- light_hue_1 2y agoThere's no mathematical reason why doing quantization after would be worse than training from scratch. That's nonsense. There's also no practical reason. Training quantized networks is often harder! This is why people quantize after the fact or do distillation. Nor is there any reason why we won't find some projection of weights onto the BitNet manifold. If it was published by academics I'd believe the cost argument. This was published by MS. They can run this experiment trivially. I have friends at MS with access to enough compute to do it in days. Either the authors ran it and saw it doesn't work or they're playing with us. Not a good look. The reviewers shouldn't have accepted the paper in this state without an explanation of why the authors can't do this. This is the question that determines if this work matters or is useless. Publishing before knowing that isn't responsible on anyone's part.
- anon373839 2y agoAfter Llama 3, does this paper’s result seem so far-fetched? That 8B parameter model showed that most of what the frontier models “know” can be represented much more compactly. So why couldn’t it be represented at low precision?