4 ms·
I have 70B at q4_0 running at 20 tokens/s inference using 2 Founders Edition RTX3090s with NVLink. It's an older AMD 5950x with a motherboard with 4-slot spacin
by jacobfnl 2y ago
I have 70B at q4_0 running at 20 tokens/s inference using 2 Founders Edition RTX3090s with NVLink. It's an older AMD 5950x with a motherboard with 4-slot spacing between the PCI-e (iirc, the motherboard is an ASRock x570 Taichi).
For comparison, I tried the same rig with a single RTX 4090, but it generated an abysmal 2.4 tokens/s, which I would consider nigh unusable. The difference between all GPU VRAM is significant.