Looks like what it does is similar to what Alex did a few years back with One Weird Trick paper: https://arxiv.org/abs/1404.5997 https://arxiv.org/abs/1404.5997

Looks like what it does is similar to what Alex did a few years back with One Weird Trick paper: https://arxiv.org/abs/1404.5997 https://arxiv.org/abs/1404.5997 When attempting to train transformers, I do notice a lot of time spend on allreduce more than with CNN models, probably due the parameter sizes. OWT seems to be natural to exploit for this situation (a lot of GEMMs, lot time spent on allreduce). Edit: Read the paper. The implementation is much less tricky than OWT, but for a good reason probably. Language model's GEMMs are smaller, therefore, partition the model would have efficiency impact (smaller GEMM will be slower). This does require much better interconnects, which NVLink / infiniband conveniently provides, that is also not available on consumer grade hardware anywhere (2-way NVLink is not meaningful).