4 ms·
The two GPUs will need to exchange information in order to complete inference, by having one GPU hold half of the network weights, and the other holding the oth
by espadrine 4y ago
The two GPUs will need to exchange information in order to complete inference, by having one GPU hold half of the network weights, and the other holding the other half. That transmission of information will be limited by the PCIe bandwidth; for instance, 30 GB/s with v4x16.
Meanwhile the A6000 VRAM bandwidth is 768 GB/s (= 16 Gb/s (GDDR6) × 384 bit-width ÷ 8 bits per byte).
- coolspot 4y agoTwo 3090 can be connected using NVLink bridge which is much faster than PCI-E.
- espadrine 4y agoI couldn’t find precise information on what bandwidth you’d get with NVLink on the 3090. To be fair, though, if all we do is inference, using Huggingface pipeline parallelism, the amount of data transferred is pretty small: 8192×2×n_tokens bytes; for most uses, with a recent PCIe setup, that bottleneck will take less than 0.1 ms per token generated, which may not be the dominant latency. (Also, the RTX 3090 has faster VRAM, >900 GB/s, than the A6000, because it is GDDR6X.)