2 ms·How are you running it locally??by arresin 2y agoHow are you running it locally??thot_experiment 2y agoI am running a 4bit imatrix quant of the 70b distill with quantized context. It fits in the 43gb of vram I have.