6 ms·
The top comment says a lot > It's pretty much the rumored size of GPT-4. However, even when quantized to 4bits, one would need ~800GB of VRAM to run it.
by karencarits 3y ago
The top comment says a lot
> It's pretty much the rumored size of GPT-4. However, even when quantized to 4bits, one would need ~800GB of VRAM to run it.
- robotnikman 3y agoWow... I won't be running anything like that locally anytime soon unfortunately...
- sp332 3y agoIt's a sparse model, which means you only need to load the "experts" that you're going to use for a given input.
- ntonozzi 3y agoBut you still need to page in the weights from disk to the GPU at each layer, right?
- juliangoldsmith 3y agoYou should only need the weights for the experts you want to run. The experts clock in at around 400 MB each (based on the 800 GB figure given elsewhere). A 24 GB GPU could fit around 60 experts, so it might be usable with a couple of old M40s.
- elorant 3y agoYou can get a second hand RTX 3090 with $800. So in theory you could build a cluster with less than $40k for all hardware. Not that this is pocket change, but given the capabilities of such a model the expense doesn’t sound that much.
- rthnbgrredf 3y agoOr one can try to rent a cluster in the clouds.
- karmasimida 3y agoyou don't have the nvlink, how would this work then
- elorant 3y agoYou don’t need the entire VRAM to be accessed as one. Everything is parallelized by the framework. You lose some speed, but it’s the only way to do it. There’s no freaking way to have 800 GB of unified VRAM.
- _zoltan_ 3y agoGH200 would like a word ;-)
- razodactyl 3y agoHahaha I love this.
- asfbnionio 3y agoAs I understand it, NVLink isn't actually required for much of anything. It allows P2P transfers and atomics. That same functionality can be used over PCIe on some GPUs with reduced performance. And the P2P features are not transparent and are lower-performance than staying inside one GPU, so generally software avoids P2P transfers even when they are available. The Nvidia documentation on this is absolutely horrible, the features vary wildly by model of GPU, and I claim no real expertise.
- htrp 3y ago3090's probably aren't gonna cut it because of the generational improvements in architecture (even if you could hack the drivers together to work)
- nullc 3y agoIt's no big deal to run an EPYC host with 2TB of ram. It'll run slow. But the Falcon 180B model runs at several tokens per second that way and its nowhere near as sparse as this should be. I also have 960GB of vram in my garage (40x P40), though I suspect getting this model distributed in a useful way might be annoying and not worth the effort particularly since it would probably be close to turn-key (if a bit slow) on a high ram epyc host.
- arthurcolle 3y agobe careful, the Compute Enforcement Administration might be reading
- crotchfire 3y agoHow many motherboards did you need to supply PCIe lanes to all of those P40s? I have a similarly-sized pile of MI25s and recently managed to get a eight of them running in a single plentiful-on-ebay supermicro motherboard (custom bifurcating fanout-riser, x8 to each card). It was a "learning experience". Falcon 180B, 4-bit quantization fits in VRAM and works, but at under 10tokens/sec because llama.cpp's support for multi-GPU on AMD is kind of an afterthought.
- nullc 3y agoI have them across 4 hosts with pci-e switches. Really only 9 per host are maximally useful due to nvidia mailbox limits in cuda (also because one per host is on 8x, IIRC).
- crotchfire 3y ago> I have them across 4 hosts with pci-e switches. Nice! Sorry to pester you, last question: did you find a source for those at a reasonable price? Pricing from last time I investigated that would've been more than the P40s, so I'm assuming you did. Apparently the actual ICs are/were incredibly expensive and only Pericom sells them as standalone chips AFAICT. (Edit: oh wow, Pericom got bought by Diodes Inc; this explains a lot) Thanks!