3 ms·
I didn't expect that, why would 4 bit be slower?
by sp332 3y ago
I didn't expect that, why would 4 bit be slower?
- ben_w 3y agoI also didn't expect it; my guess is that the library is doing 8 (or more) bit arithmetic, and using bit masking and shifting to un/compress the values?
- qeternity 3y agoIt isn't. It's just that 4bit implementations are really bad. Exllama has shown how properly optimized and fused kernels can perform.