3 ms·
Curious to see how it performs against GPT-4. Mixtral8x22 beats CommandR+, which is at GPT-4-level in LMSYS' leaderboard.
by arnaudsm 3y ago
Curious to see how it performs against GPT-4.
Mixtral8x22 beats CommandR+, which is at GPT-4-level in LMSYS' leaderboard.
- zone411 2y agoLMSYS leaderboard is just one benchmark (that I think is fundamentally flawed). GPT-4 is clearly better.
- arnaudsm 2y agoWhich alterative benchmarks do you recommend?
- zone411 2y agoI don't have a perfect solution, except for the obvious answer that the best we can do is a combination of multiple benchmarks. It's harder now than ever because you also want to test long contexts, and older benchmarks are over-optimized. I think it would be best to overweigh newer benchmarks and underweigh benchmarks on which objectively dumb models like Claude 3 Haiku score well. And I have a big problem with HumanEval, which is Python-only and has only 164 problems but is used as the catch-all for coding abilities.