Batches need to be reasonably sized. Llms are usually trained for a single epoch. If you just average the gradients once, it will be awful. The main problem I

Batches need to be reasonably sized. Llms are usually trained for a single epoch. If you just average the gradients once, it will be awful. The main problem I believe is that you would need extremely high bandwidth. For a 100B model you're looking at 400GB of gradients you need to send each batch. There's a reason why super fast networking is used.