4 ms·
The combined rankings on HF aren't just the arena scores, and certainly MMLU is an example of Goodhart's Law at this point. The Chatbot arena is more an issue
by kromem 3y ago
The combined rankings on HF aren't just the arena scores, and certainly MMLU is an example of Goodhart's Law at this point.
The Chatbot arena is more an issue of sampling bias, and I think it would be pretty interesting to run an analysis of random samples on the prompts provided to see just how broad they are or aren't.
- maeil 3y agoIt is trivially easy to see just how massive the sampling bias is. There are countless tasks where e.g. the gap between a version of Mistral and a version of GPT are incredibly large. Anything that requires less common knowledge, anything creative in a random language, let's say Bulgarian, or Thai. Yet on the leaderboard things are very different, because the arena is only used by developers who verify performance on tech-related prompts in English.