3 ms·
These models aren't even trained on FP32. The original format is FP16. And quantizing to INT8/FP8 is almost lossless. But yes, 2.5 bits per weight is pretty in
by redox99 3y ago
These models aren't even trained on FP32. The original format is FP16. And quantizing to INT8/FP8 is almost lossless.
But yes, 2.5 bits per weight is pretty insane.