4 ms·
There's no rhyme or reason to it. Quants aren't benchmarked much. Generally 4bit better than smaller model 8bit
by markasoftware 2mo ago
There's no rhyme or reason to it. Quants aren't benchmarked much. Generally 4bit better than smaller model 8bit