2 ms·
It's no big deal to run an EPYC host with 2TB of ram. It'll run slow. But the Falcon 180B model runs at several tokens per second that way and its nowhere near
by nullc 3y ago
It's no big deal to run an EPYC host with 2TB of ram. It'll run slow. But the Falcon 180B model runs at several tokens per second that way and its nowhere near as sparse as this should be.
I also have 960GB of vram in my garage (40x P40), though I suspect getting this model distributed in a useful way might be annoying and not worth the effort particularly since it would probably be close to turn-key (if a bit slow) on a high ram epyc host.
- arthurcolle 3y agobe careful, the Compute Enforcement Administration might be reading
- crotchfire 3y agoHow many motherboards did you need to supply PCIe lanes to all of those P40s? I have a similarly-sized pile of MI25s and recently managed to get a eight of them running in a single plentiful-on-ebay supermicro motherboard (custom bifurcating fanout-riser, x8 to each card). It was a "learning experience". Falcon 180B, 4-bit quantization fits in VRAM and works, but at under 10tokens/sec because llama.cpp's support for multi-GPU on AMD is kind of an afterthought.
- nullc 3y agoI have them across 4 hosts with pci-e switches. Really only 9 per host are maximally useful due to nvidia mailbox limits in cuda (also because one per host is on 8x, IIRC).
- crotchfire 3y ago> I have them across 4 hosts with pci-e switches. Nice! Sorry to pester you, last question: did you find a source for those at a reasonable price? Pricing from last time I investigated that would've been more than the P40s, so I'm assuming you did. Apparently the actual ICs are/were incredibly expensive and only Pericom sells them as standalone chips AFAICT. (Edit: oh wow, Pericom got bought by Diodes Inc; this explains a lot) Thanks!