4 ms·
What was the main focus when training this model? Besides the ELO score, it's looking like the models (31B / 26B-A4) are underperforming on some of the typical
by _boffin_ 6mo ago
What was the main focus when training this model? Besides the ELO score, it's looking like the models (31B / 26B-A4) are underperforming on some of the typical benchmarks by a wide margin. Do you believe there's an issue with the tests or the results are misleading (such as comparative models benchmaxxing)?
Thank you for the release.
- BoorishBears 6mo agoBecnhmarks are a pox on LLMs. You can use this model for about 5 seconds and realize its reasoning is in a league well above any Qwen model, but instead people assume benchmarks that are openly getting used for training are still relevant.
- j45 6mo agoDefinitely have to use each model for your use case personally, many models can train to perform better on these tests but that might not transfer to your use case.
- girvo 6mo agoThey really are. Benchmaxxing is real… but also the Qwen 3.5 series of models are still very impressive. I’m looking forward to trying out Gemma