The gradient info can be compressed 10000x with the right tricks, I think it is achievable. Nous claims they did it already: https://github.com/NousResearch/Di

The gradient info can be compressed 10000x with the right tricks, I think it is achievable. Nous claims they did it already: https://github.com/NousResearch/DisTrO https://github.com/NousResearch/DisTrO There are other gradient compression papers from the past reporting large compression rates