4 ms·
My take: the distills under 32B aren’t worth running. Quants seem to impact quality much more than other models. 32B and 70B unquantized are very good. 671B is
by deoxykev 2y ago
My take: the distills under 32B aren’t worth running. Quants seem to impact quality much more than other models. 32B and 70B unquantized are very good. 671B is SOTA.
- rom16384 2y agoIn my own tests even the 70B distill has an unacceptably high rate of hallucinations that makes it hard to trust the results.