See also zeroth order backpropagation which allows 300X faster training while not reducing throughput that much https://arxiv.org/abs/2011.08895 https://arxiv.o

See also zeroth order backpropagation which allows 300X faster training while not reducing throughput that much https://arxiv.org/abs/2011.08895 https://arxiv.org/abs/2011.08895 How much zero-3 affect accuracy? See also https://github.com/microsoft/fastformers https://github.com/microsoft/fastformers