4 ms·
Yeah, you need 100gbe minimal. 10gbe is too little (PCIe bandwidth can be a bottleneck, and that is already clocked around 100GbE (16GB)). BTW: echo to the aut
by liuliu 3y ago
Yeah, you need 100gbe minimal. 10gbe is too little (PCIe bandwidth can be a bottleneck, and that is already clocked around 100GbE (16GB)).
BTW: echo to the author, PSU and in the U.S. (120v) is a major issue why I am limiting to 4-GPUs. Also, it seems 3090 still have NVLink support, wondering why the author haven't put that up. From what I experienced, NVLink does help if you run data parallel training.
- jacquesm 3y agoCouldn't you use a 240V dryer socket for that purpose? That should get you 7200 Watts on a 30A circuit.
- supertrope 3y agoYou might need a second 240V outlet for the air conditioner to evacuate that much heat.
- jacquesm 3y agoTrue, but when you operate 7 GPUs on one board I reckon keeping a good eye on your thermals is where it starts. Otherwise you can kiss your GPUs goodbye well before you reach the payback moment. Edit: and you'd have to factor in that cooling power as well into the running costs.
- deleted 3y ago[deleted]
- buildbot 3y agoThey might not be using nvlink due to selling on vast, someone could rent a single GPU out of the four available. No idea if there is some cross user security implication with nvlink in that scenario.
- choppaface 3y agoDo you have any examples of nvlink improving performance for 3090 with vanilla pytorch DDP? Or are you talking some other training impl?
- Tostino 3y agoI've literally not seen any real (more than buying another card) improvement with NVlink on anything i've seen online over the past ~year. I'd honestly be a bit suspect.
- choppaface 3y agoI have seen some benefit on A100s only but am unfamiliar with how to get any nvlink gains on RTX cards. Monitoring vanilla pytorch DDP in nvtop on RTX, I haven’t seen PCIE bus transfer speeds approaching the theoretical max. The OP uses bifurcators in his 8-gpu box so clearly OP does not seem to be bus-limited.
- liuliu 3y agoYeah, I am talking about ~10% performance differences for a specific model (I believe it is when benchmarking vanilla seq2seq model (from AIAYN paper) with small batch size and no gradient accumulation). NVLink is about ~$100, so for these cases, I guess you shouldn't expect it to be "more than buying another card" type of improvements.
- liuliu 3y agoIt is model-dependent. I've seen that (NVLink benefits) when comparing against PCIe-3 connection, with small batch size, no gradient accumulation. Once you have larger batch size and gradient accumulation, DDP won't be improved by NVLink I believe (the all-reduce traffic on gradients will be small comparing to your computation overhead).
- wredue 3y agoI would be additionally skeptical that there’s any consumer, or even fire decal hardware (gamer brand stuff) that actually delivers consistently on 10gbe across several nodes. Enthusiast / small business / entry level enterprise gear will get you there, but you’re looking at several hundred dollars per port.
- magixx 3y ago10Gbe can be pretty cheap ($30 for X540) PC side with used hardware (SFP mostly and not multigig however). Even generic PCIe cards can add a RJ45 10G multigig port for ~$50. The switch/router side is where I find it gets expensive.
- nixgeek 3y agoYou can do 100GbE for about $150 a port in switch cost (new); you sometimes see ConnectX-5 cards on eBay for about $100-150 a port (used). I’ve got a fairly good amount of 100GbE in my homelab. Pretty affordable in 2023. 200GbE and 400GbE is still totally unaffordable for anything remotely personal, IMO.