3 ms·
Stupid question: how can you do deep learning on 1 bit? DL works by calculating a gradient and following it to minimize a loss function, but if all your paramet
by littlestymaar 3y ago
Stupid question: how can you do deep learning on 1 bit? DL works by calculating a gradient and following it to minimize a loss function, but if all your parameters are 1 bit, there's no such thing as a gradient as you don't have differentiation at all. So how does it work?
- edflsafoiewq 3y agoNot sure what you mean, why wouldn't there be a gradient? There is a problem that small changes to weights may have no effect (eg ((0 + 0.3) + 0.3) + 0.4 = 0 if you have to round the result to 0 or 1 after every step), but to fix this I believe everyone maintains high precision weights for the backwards pass, from which the quantized weights are derived.
- scotty79 3y agoYou could treat fractional addition as probabilistic operation.
- Centigonal 3y agoNot a stupid question at all. Early quantization approaches just quantized a high-bit-precision pre-trained model - no need to calculate gradients on the quantized weights. BitNet[1] changed the game by training a low-precision model from scratch. It achieves this by keeping high precision in the gradients, optimizer state, and in "latent weights" which are then quantized on the fly. I don't really understand the finer details of how this works, so check out the paper if you're interested. This article's approach is interesting. They start by quantizing a pre-trained high-precision model, and then they fine-tune the quantized model using LoRA (which dramatically improves the performance of the quantized model). They don't talk about the bit depth of the values in the LoRA matrices, so it may be that those are higher bit-depth. [1] https://arxiv.org/pdf/2310.11453.pdf https://arxiv.org/pdf/2310.11453.pdf