The original bitnet paper has more information on training: https://arxiv.org/abs/2310.11453 https://arxiv.org/abs/2310.11453 . >To train our 1-bit model, we e

The original bitnet paper has more information on training: https://arxiv.org/abs/2310.11453 https://arxiv.org/abs/2310.11453 . >To train our 1-bit model, we employ the straight-through estimator (STE)[BLC13] to approximate the gradient during backpropagation. This method bypasses the nondifferentiable functions, such as the Sign (Eq. 2) and Clip (Eq. 5) functions, during the backward pass. STE allows gradients to flow through the network without being affected by these non-differentiable functions, making it possible to train our quantized model. >While the weights and the activations are quantized to low precision, the gradients and the optimizer states are stored in high precision to ensure training stability and accuracy. Following the previous work [LSL+21], we maintain a latent weight in a high-precision format for the learnable parameters to accumulate the parameter updates. The latent weights are binarized on the fly during the forward pass and never used for the inference process.