2 ms·
Have you compared it to the 1.58 bit dynamic quant model based on the original R1 (i.e., not a distillation)? Whatever unsloth did, it doesn't seem to be givin
by CamperBob2 2y ago
Have you compared it to the 1.58 bit dynamic quant model based on the original R1 (i.e., not a distillation)? Whatever unsloth did, it doesn't seem to be giving up much reasoning performance over the full Q8 version.
- mluo 2y agoIt's simply bc the model is small (1.5B), making it sensitive to weight perturbations