3 ms·

Maybe it’s to speed up multi gpu matrix multiplies. They’re useful for serving/training gpt3 size models