4 ms·
How does backprop work here? I can't imagine flipping bits of everything upstream of an error is effective.
by stormfather 3y ago
How does backprop work here? I can't imagine flipping bits of everything upstream of an error is effective.
- spyder 3y agoFrom the BitNet paper: "Straight-through estimator. To train our 1-bit model, we employ the straight-through estimator (STE)[BLC13] to approximate the gradient during backpropagation. This method bypasses the nondifferentiable functions, such as the Sign (Eq. 2) and Clip (Eq. 5) functions, during the backward pass. STE allows gradients to flow through the network without being affected by these non-differentiable functions, making it possible to train our quantized model." also the author's (@shumingma) answer in the comments: https://huggingface.co/papers/2402.17764#65df17ed4d436404cdc7b34a https://huggingface.co/papers/2402.17764#65df17ed4d436404cdc...
- joelthelion 3y ago(haven't read the paper). Maybe you can flip bits with a probability distribution that depends on the gradient?
- stormfather 3y agoThat's an interesting idea! Would love to try that on MNIST one day.