3 ms·
Do you have any examples of nvlink improving performance for 3090 with vanilla pytorch DDP? Or are you talking some other training impl?
by choppaface 3y ago
Do you have any examples of nvlink improving performance for 3090 with vanilla pytorch DDP? Or are you talking some other training impl?
- Tostino 3y agoI've literally not seen any real (more than buying another card) improvement with NVlink on anything i've seen online over the past ~year. I'd honestly be a bit suspect.
- choppaface 3y agoI have seen some benefit on A100s only but am unfamiliar with how to get any nvlink gains on RTX cards. Monitoring vanilla pytorch DDP in nvtop on RTX, I haven’t seen PCIE bus transfer speeds approaching the theoretical max. The OP uses bifurcators in his 8-gpu box so clearly OP does not seem to be bus-limited.
- liuliu 3y agoYeah, I am talking about ~10% performance differences for a specific model (I believe it is when benchmarking vanilla seq2seq model (from AIAYN paper) with small batch size and no gradient accumulation). NVLink is about ~$100, so for these cases, I guess you shouldn't expect it to be "more than buying another card" type of improvements.
- liuliu 3y agoIt is model-dependent. I've seen that (NVLink benefits) when comparing against PCIe-3 connection, with small batch size, no gradient accumulation. Once you have larger batch size and gradient accumulation, DDP won't be improved by NVLink I believe (the all-reduce traffic on gradients will be small comparing to your computation overhead).