4 ms·
> This is because without dropping serious $$$ on mellanox high-speed NICs and switches, inter-server communication bandwidth quickly becomes the bottleneck whe
by throwing_away 3y ago
> This is because without dropping serious $$$ on mellanox high-speed NICs and switches, inter-server communication bandwidth quickly becomes the bottleneck when training large models. I can’t afford fancy enterprise grade hardware, so I get around it by keeping my compute all on the same machine. This goal drives many of the choices I made in building out my servers, as you will see.
10gbe is very cheap now, but I guess that's not enough?
- liuliu 3y agoYeah, you need 100gbe minimal. 10gbe is too little (PCIe bandwidth can be a bottleneck, and that is already clocked around 100GbE (16GB)). BTW: echo to the author, PSU and in the U.S. (120v) is a major issue why I am limiting to 4-GPUs. Also, it seems 3090 still have NVLink support, wondering why the author haven't put that up. From what I experienced, NVLink does help if you run data parallel training.
- jacquesm 3y agoCouldn't you use a 240V dryer socket for that purpose? That should get you 7200 Watts on a 30A circuit.
- supertrope 3y agoYou might need a second 240V outlet for the air conditioner to evacuate that much heat.
- jacquesm 3y agoTrue, but when you operate 7 GPUs on one board I reckon keeping a good eye on your thermals is where it starts. Otherwise you can kiss your GPUs goodbye well before you reach the payback moment. Edit: and you'd have to factor in that cooling power as well into the running costs.
- deleted 3y ago[deleted]
- buildbot 3y agoThey might not be using nvlink due to selling on vast, someone could rent a single GPU out of the four available. No idea if there is some cross user security implication with nvlink in that scenario.
- choppaface 3y agoDo you have any examples of nvlink improving performance for 3090 with vanilla pytorch DDP? Or are you talking some other training impl?
- Tostino 3y agoI've literally not seen any real (more than buying another card) improvement with NVlink on anything i've seen online over the past ~year. I'd honestly be a bit suspect.
- choppaface 3y agoI have seen some benefit on A100s only but am unfamiliar with how to get any nvlink gains on RTX cards. Monitoring vanilla pytorch DDP in nvtop on RTX, I haven’t seen PCIE bus transfer speeds approaching the theoretical max. The OP uses bifurcators in his 8-gpu box so clearly OP does not seem to be bus-limited.
- liuliu 3y agoYeah, I am talking about ~10% performance differences for a specific model (I believe it is when benchmarking vanilla seq2seq model (from AIAYN paper) with small batch size and no gradient accumulation). NVLink is about ~$100, so for these cases, I guess you shouldn't expect it to be "more than buying another card" type of improvements.
- liuliu 3y agoIt is model-dependent. I've seen that (NVLink benefits) when comparing against PCIe-3 connection, with small batch size, no gradient accumulation. Once you have larger batch size and gradient accumulation, DDP won't be improved by NVLink I believe (the all-reduce traffic on gradients will be small comparing to your computation overhead).
- wredue 3y agoI would be additionally skeptical that there’s any consumer, or even fire decal hardware (gamer brand stuff) that actually delivers consistently on 10gbe across several nodes. Enthusiast / small business / entry level enterprise gear will get you there, but you’re looking at several hundred dollars per port.
- magixx 3y ago10Gbe can be pretty cheap ($30 for X540) PC side with used hardware (SFP mostly and not multigig however). Even generic PCIe cards can add a RJ45 10G multigig port for ~$50. The switch/router side is where I find it gets expensive.
- nixgeek 3y agoYou can do 100GbE for about $150 a port in switch cost (new); you sometimes see ConnectX-5 cards on eBay for about $100-150 a port (used). I’ve got a fairly good amount of 100GbE in my homelab. Pretty affordable in 2023. 200GbE and 400GbE is still totally unaffordable for anything remotely personal, IMO.
- bradfox2 3y ago100gbe mellanox connect x cards are not actually that expensive though.
- jacquesm 3y agoYou'd need a switch too, unless you're going point-to-point but that will eat up PCI slots that you probably would like to use for GPUs.
- tuetuopay 3y agomikrotik makes some pretty nice hardware for rather cheap. they now offer a 4x100Gbps switch for $800, which is a darn good price.
- jacquesm 3y agoIndeed, at $200 / port that's really neat if that's all you need. Funny, I remember paying close to $1000 / port for 100 mbps not all that long ago :)
- bradfox2 3y agoYou really want gpu rdma though. It's a bit of a pain to get setup but it's worth it.
- jacquesm 3y agoNot on the 3090 though, isn't it?
- LTL_FTC 3y agoIf these server boards support thunderbolt AIC's, and I believe they might as my Threadripper Pro board does, daisy chaining them together could get you 40Gbps somewhat easily, if that is sufficient.