7 ms·
tinygrad supports uneven splits. There's no fundamental reason for 4 or 8, and work should almost fully parallelize on any number of GPUs with good software. W
by georgehotz 3y ago
tinygrad supports uneven splits. There's no fundamental reason for 4 or 8, and work should almost fully parallelize on any number of GPUs with good software.
We chose 6 because we have 128 PCIe lanes, aka 8 16x ports. We use 1 for NVMe and 1 for networking, leaving 6 for GPUs to connect them in full fabric. If we used 4 GPUs, we'd be wasting PCIe, and if we used 8 there would be no room for external connectivity aside from a few USB3 ports.
- doctorpangloss 3y agoHave you compared 3x 3090-3090 pairs over NVLink? IMO the most painful thing is that since these hardware configurations are esoteric, there is no software that detects them and moves things around "automatically." Regardless of what people thing device_map="auto" does, and anyway, Hugging Face's transformers/diffusers are all over the place.
- davidzweig 3y agoIs it possible a similar patch would work for P2P on 3090s? btw, I found a Gigabyte board on Taobao that is unlisted on their site: MZF2-AC0, costs $900. 2 socket Epyc and 10 PCIE slots, may be of interest. A case that should fit, with 2x 2000W Great Wall PSUs and PDU is 4050 RMB (https://www.toploong.com/en/4GPU-server-case/644.html https://www.toploong.com/en/4GPU-server-case/644.html). You still need blower GPUs.
- georgehotz 3y agoIt should if your 3090s have Resizable BAR support in the VBIOS. AFAIK most card manufacturers released BIOS updates enabling this. Re: 3090 NVLink, that only allows pairs of cards to be connected. PCIe allows full fabric switch of many cards.
- learnedmachine 3y agoThanks for the amazing work! I tried the driver with some 3090s (all of which show the 32G line with lspci -s 01:00.0 -v) and while torch says I have p2p access, I can't get it to work with anything as I get illegal memory access errors.
- Ratiofarmings 3y agoIn cases where they didn't, the techpowerup vBIOS collection solves the problem.
- cjbprime 3y agoDoesn't nvlink work natively on 3090s? I thought it was only removed (and here re-enabled) in 4090.
- qeternity 3y agoThis not not nvlink.
- deleted 3y ago[deleted]
- davidzweig 3y agoUpdate, I checked with the case company, toploong, they say that board is a 5mm too big or so for the case.
- andersa 3y agoThat is very interesting if tinygrad can support it! Every other library I've seen had the limitation on dividing the heads, so I'd (perhaps incorrectly) assumed that it's a general problem for inference.
- spmurrayzzz 3y agoThere are some interesting hacks you can do like replicating the K/V weights by some factor which allows them to be evenly divisible by whatever number of gpus you have. Obviously there is a memory cost there, but it does work.
- Emerson1 2y agohow could you go about testing this on say llama3 70b with two 4090s - vllm supports tensor parallelism, would the expectation be that inference would be faster with p2p? How would you update the nvidia driver tho? Thanks any thought appreciated
- AnthonyMouse 3y agoIs there any reason you couldn't use 7? 8 PCIe lanes each seems more than sufficient for NVMe and networking.
- WanderPanda 3y agoDid you at least front run the market and stocked up of 4090ies before this release? Also gamers are probably not too happy about these developments :D
- tinco 3y ago4090's have consistently been around 2000 dollars. I don't think there's many gamers who would be affected by price fluctuations of the 4090 or even the 4080.
- presides 3y agoThis is out of touch; they were mad before and they will be mad again. Lots of people spend a huge chunk of their modest disposable income on high end gaming gear, and the only upside of these issues for them is that eventually, YEARS down the line, capacity/supply issues MIGHT calm down in a way that yields some benefits. They're going to realize soon enough that they've basically just been told that the extremely shitty problem they thought they'd moved beyond is back with a vengeance and the next generation of gaming cards has the potential to make the past few rounds of scalping shit-shows look tame.
- swalsh 3y agoGamers have a TON of really good really affordable options. But you kind of need 24gb min unless you're using heavy quantization. So 3090 and 4090's are what local llm people are building with (mostly 3090's as you can get then for about $700, and they're dang good)