4 ms·
The 70b distill at 4bit quantize fits, so yes, and performance and quality seem pretty good. I can't run the gigantic one.
by api 2y ago
The 70b distill at 4bit quantize fits, so yes, and performance and quality seem pretty good. I can't run the gigantic one.
- azinman2 2y agoOk but that’s not deepseek-r1. Lots of people keep saying this for distills and it’s getting very confusing.