SOTA on the biggest language models (which is where effectively the largest models are) is here: https://www.microsoft.com/en-us/research/blog/zero-infinity-and

SOTA on the biggest language models (which is where effectively the largest models are) is here: https://www.microsoft.com/en-us/research/blog/zero-infinity-and-deepspeed-unlocking-unprecedented-model-scale-for-deep-learning-training/ https://www.microsoft.com/en-us/research/blog/zero-infinity-...