4 ms·
It got me thinking, what might it look like to natively train a binary quantized embedding model? You can’t do calculus per se on {0,1}, but maybe you could do
by kevinventullo 2y ago
It got me thinking, what might it look like to natively train a binary quantized embedding model? You can’t do calculus per se on {0,1}, but maybe you could do something like randomly flip bits with a probability weighted by the severity of the error during backprop… anyway, I’m sure there’s plenty of literature about this.
- alok-g 2y agoYou may find this interesting: https://writings.stephenwolfram.com/2024/08/whats-really-going-on-in-machine-learning-some-minimal-models/ https://writings.stephenwolfram.com/2024/08/whats-really-goi...
- janalsncm 2y agoProbably just an extreme version of quantization-aware training? During training you round the prediction to the range you want, but keep it as a float. Since rounding isn’t differentiable there’s fancy techniques to approximate that as well. > QAT backward pass typically uses straight-through estimators (STE), a mechanism to estimate the gradients flowing through non-smooth functions https://pytorch.org/blog/quantization-aware-training/ https://pytorch.org/blog/quantization-aware-training/