4 ms·
Preliminary analysis via lm-evaluation-harness + vllm model | disk | wikitext | gsm8k (match/error) baseline | 55G | 8.00 | 0.50/0.09
by verdverm 4mo ago
Preliminary analysis via lm-evaluation-harness + vllm
model | disk | wikitext | gsm8k (match/error)
baseline | 55G | 8.00 | 0.50/0.09
nvfp4-gptq | 27G | 8.25 | 0.47/0.9
nvfp4a16-gptq | 27G | 8.11 | 0.53/0.9
bonsai-4bit | 19G | 16.75 | 0/0 (eval bug?)
Looks like they quant'd too hard at 4 bits, can't imagine the ternary being any good based on this. I'm also not sure what is up with the gsm8k, their benchmarks show something different, but they are using another eval tool. I'll have to add it to my setup. Also why I'm building a setup instead of taking model devs word for benchmarks. (https://github.com/modelscope/evalscope https://github.com/modelscope/evalscope)
Code if you'd like to reproduce or try other test sets: https://github.com/verdverm/quantr https://github.com/verdverm/quantr (lightly tuned to a single oem spark, probably possible in 32-48G)
Good paper to understand the effects of quant regimes across model families and tasks: https://arxiv.org/abs/2402.18158 https://arxiv.org/abs/2402.18158 (Evaluating Quantized Large Language Models - 2024 ICML)
- kamranjon 4mo agoDoesn’t this suggest you aren’t properly running the model?
- verdverm 4mo agoIts unclear if its in vllm, the eval harness, or the weights. I'm running it the same as other qwen3.6 derived models and it appears to work, but other comments speak of waiting for support in their tool. Could be a chat template or could be legit because I'm using a different quant / bit selection from their hugging face and using their primary model as the comparison point for my huh? They may have put more effort into the flagship model than the 4bit because they are focused on on-device model running. Software was already complex, now we are adding highly non deterministic elements into the mix