4 ms·
So are there any details on the algorithms they used for backprop? I'm not seeing any in the paper other than "we used a lot of tokens".
by llm_trw 3y ago
So are there any details on the algorithms they used for backprop? I'm not seeing any in the paper other than "we used a lot of tokens".
- IanCal 3y agoDoes this help? https://arxiv.org/abs/2310.11453 https://arxiv.org/abs/2310.11453 It seems to have more details (it's the paper before the linked one) about the actual training, but I'm scanning it and this isn't my field so maybe it's too light also.
- llm_trw 3y agoNot really, that's for the binary version of the algorithm, the ternary version can propagate a lot more information in the backwards pass using the fact outputs either -1, 0, 1. But I imagine they are using the same thing since a bunch of the authors are the same.
- wongarsu 3y agoIt's a fairly straightforward modification of BitNet, so I assume this quote from the BitNet paper applies: To train our 1-bit model, we employ the straight-through estimator (STE)[BLC13 ] to approximate the gradient during backpropagation. This method bypasses the non-differentiable functions, such as the Sign (Eq. 2) and Clip (Eq. 5) functions, during the backward pass. STE allows gradients to flow through the network without being affected by these non-differentiable functions, making it possible to train our quantized model