3 ms·
I run q5s usually, since it's a 40% haircut on model size, with nearly no PPL loss. (Presuming an 8b native model like Qwen.)
by a_wild_dandan 2y ago
I run q5s usually, since it's a 40% haircut on model size, with nearly no PPL loss. (Presuming an 8b native model like Qwen.)