3 ms·
Has anyone published a similar run of benchmarks with llama2 70B but at different quantization levels? I assume this benchmark is evaluated on the base model r
by mk_stjames 3y ago
Has anyone published a similar run of benchmarks with llama2 70B but at different quantization levels? I assume this benchmark is evaluated on the base model run at FP16. How much does it lose quantizing to INT8?