3 ms·
I am running a 4bit imatrix quant of the 70b distill with quantized context. It fits in the 43gb of vram I have.
by thot_experiment 2y ago
I am running a 4bit imatrix quant of the 70b distill with quantized context. It fits in the 43gb of vram I have.