3 ms·I would look into running a 4 bit quant using llama cpp (or any of its wrappers)by pixelmelt 9mo agoI would look into running a 4 bit quant using llama cpp (or any of its wrappers)