I'm impressed by how many of the new benchmarks that the Qwen team ran. As the old benchmarks get saturated/overfit, new ones are of course required. Some of th

I'm impressed by how many of the new benchmarks that the Qwen team ran. As the old benchmarks get saturated/overfit, new ones are of course required. Some of the latest ones they use include: * MMLU-Pro https://github.com/TIGER-AI-Lab/MMLU-Pro https://github.com/TIGER-AI-Lab/MMLU-Pro - a new more challenging (and improved in other areas) version of MMLU that does a better job separating out the current top models * MixEval(-Hard) https://github.com/Psycoy/MixEval https://github.com/Psycoy/MixEval - a very quick/cheap eval that has high correlation w/ Chatbot Arena ELOs that can w/ (statistically correlated) dynamically swappable question sets * Arena Hard https://github.com/lm-sys/arena-hard-auto https://github.com/lm-sys/arena-hard-auto - another automatic eval tool that uses LLM-as-a-Judge w/ high correlation w/ Chatbot Arena / human rankings * LiveCodeBench https://livecodebench.github.io/ https://livecodebench.github.io/ - a coding test with different categories based off of LeetCode problems that also lets you filter/compare scores by problem release month to see if the impact of overfitting/contamination