Microsoft Research published results with 1-bit gradients. https://www.microsoft.com/en-us/research/publication/1-bit-stochastic-gradient-descent-and-applicati

Microsoft Research published results with 1-bit gradients. https://www.microsoft.com/en-us/research/publication/1-bit-stochastic-gradient-descent-and-application-to-data-parallel-distributed-training-of-speech-dnns/ https://www.microsoft.com/en-us/research/publication/1-bit-s... int4 (fixed point) has already been popular for inference https://developer.nvidia.com/blog/int4-for-ai-inference/ https://developer.nvidia.com/blog/int4-for-ai-inference/ and int3 has seem some use for LLaMA-at-home