6 ms·
With 16gb you can comfortably run a 12b model that's been quantized. Quantizing is (bad example) effectively lossy compression.
by hadlock 1y ago
With 16gb you can comfortably run a 12b model that's been quantized. Quantizing is (bad example) effectively lossy compression.