3 ms·

I got it from section 2.2 > The number of model parameters is slightly higher in the BitLinear setting, as we both have 1.58-bit weights as well as the 16-bit

I got it from section 2.2 > The number of model parameters is slightly higher in the BitLinear setting, as we both have 1.58-bit weights as well as the 16-bit shadow weights. However, this fact does not change the number of trainable/optimized parameters in practice. https://arxiv.org/html/2407.09527v1 https://arxiv.org/html/2407.09527v1

buildbot 2y ago
Exactly as xnornet was doing way back in 2016 - shadow 32bit weights, quantized to 1 bit during the forward pass. https://arxiv.org/abs/1603.05279 https://arxiv.org/abs/1603.05279 I personally have a pretty negative opinion of the bitnet paper.
- llm_trw 2y ago
  Thanks for the citation, I did my work in the area around 2014 and never looked back. That's a very good summary of the state of the field as I remember it.
sigmoid10 2y ago
What? That's the wrong paper. It is not even from Microsoft. This is it: https://www.microsoft.com/en-us/research/publication/bitnet-scaling-1-bit-transformers-for-large-language-models/ https://www.microsoft.com/en-us/research/publication/bitnet-... >we introduce BitLinear as a drop-in replacement of the the nn.Linear layer in order to train 1-bit weights from scratch
- llm_trw 2y ago
  Section 2.2 from your paper, with less clarity and more obfuscation: >While the weights and the activations are quantized to low precision, the gradients and the optimizer states are stored in high precision to ensure training stability and accuracy. Following the previous work [ LSL+21 ], we maintain a latent weight in a high-precision format for the learnable parameters to accumulate the parameter updates. The latent weights are binarized on the fly during the forward pass and never used for the inference process. https://arxiv.org/pdf/2310.11453 https://arxiv.org/pdf/2310.11453 The other paper had a much nicer and clearer introduction to bitlinear than the original Microsoft paper, which is why I used it. Uncharitably you might say that they aren't hiding the lead 10 paragraphs in.
  - sigmoid10 2y ago
    They are not hiding anything, because this is standard behaviour for all current optimisers. You still get a massive memory improvement from lower bit model weights during training.
    - llm_trw 2y ago
      I'm sorry but you just don't understand what the paper is saying.

buildbot 2y ago

Exactly as xnornet was doing way back in 2016 - shadow 32bit weights, quantized to 1 bit during the forward pass. https://arxiv.org/abs/1603.05279 https://arxiv.org/abs/1603.05279 I personally have a pretty negative opinion of the bitnet paper.

llm_trw 2y ago
Thanks for the citation, I did my work in the area around 2014 and never looked back. That's a very good summary of the state of the field as I remember it.

sigmoid10 2y ago

What? That's the wrong paper. It is not even from Microsoft. This is it: https://www.microsoft.com/en-us/research/publication/bitnet-scaling-1-bit-transformers-for-large-language-models/ https://www.microsoft.com/en-us/research/publication/bitnet-... >we introduce BitLinear as a drop-in replacement of the the nn.Linear layer in order to train 1-bit weights from scratch

llm_trw 2y ago
Section 2.2 from your paper, with less clarity and more obfuscation: >While the weights and the activations are quantized to low precision, the gradients and the optimizer states are stored in high precision to ensure training stability and accuracy. Following the previous work [ LSL+21 ], we maintain a latent weight in a high-precision format for the learnable parameters to accumulate the parameter updates. The latent weights are binarized on the fly during the forward pass and never used for the inference process. https://arxiv.org/pdf/2310.11453 https://arxiv.org/pdf/2310.11453 The other paper had a much nicer and clearer introduction to bitlinear than the original Microsoft paper, which is why I used it. Uncharitably you might say that they aren't hiding the lead 10 paragraphs in.
- sigmoid10 2y ago
  They are not hiding anything, because this is standard behaviour for all current optimisers. You still get a massive memory improvement from lower bit model weights during training.
  - llm_trw 2y ago
    I'm sorry but you just don't understand what the paper is saying.