3 ms·
A 4090 will just narrowly fit a 34B parameter model at 4-bit quantisation. A 12B model will run on a 4090 with plenty room to spare, even with 8-bit quantisati
by michaelt 2y ago
A 4090 will just narrowly fit a 34B parameter model at 4-bit quantisation.
A 12B model will run on a 4090 with plenty room to spare, even with 8-bit quantisation.