3 ms·
12Gb of VRAM lets you run 13B models (4-bit quantized) with reasonable speed, and can be had for under $300 if you go for previous-generation NVidia hardware. P
by int_19h 3y ago
12Gb of VRAM lets you run 13B models (4-bit quantized) with reasonable speed, and can be had for under $300 if you go for previous-generation NVidia hardware. Plenty of developers around with M1 and M2 Macs, as well.