I don't use NCCL, I wrote my own ring collectives after reinventing them in 2014 (I felt really clever for a whole 20 minutes before a Google search popped my b

I don't use NCCL, I wrote my own ring collectives after reinventing them in 2014 (I felt really clever for a whole 20 minutes before a Google search popped my balloon) to avoid working with a vendor that was trying to gouge us on servers: they beat NCCL by ~50%. That said, most people couldn't write them so I advise NCCL. You're the third person to tell me NCCL has ish(tm), fascinating. And sure, you could do the outer product trick. You could use non-deterministic ASGD. And you can do a lot of weird(tm) tricks. But why do these (IMO ad-hoc and potentially sketchy task-specific) things when there's an efficient way to parallelize the original network in a deterministic manner for training that allows performance on par with a DGX-1 server? Because for me, the action is in automagically discovering the optimal deterministic distribution of the computation so researchers don't need to worry about it. And IMO that's where the frameworks fail currently.