4 ms·
No one runs inference at full FP16 precision. 6 bit quant gives ~90% of original quality at more than half memory savings.
by stevenhuang 3y ago
No one runs inference at full FP16 precision. 6 bit quant gives ~90% of original quality at more than half memory savings.