I think most open source LLM leaderboards are bad - I wish they used different benchmarks to rank the LLMs. The benchmarks they use test metrics that are irrele

I think most open source LLM leaderboards are bad - I wish they used different benchmarks to rank the LLMs. The benchmarks they use test metrics that are irrelevant to most users. Also, if you've ever looked inside the benchmark datasets (I have) - it's not the stuff that I'd want dictating the rankings on the LLM leaderboards that influences what LLMs people use! I have ideas on making better benchmarks (in short: take actual ChatGPT session history, and actual LLM API production use queries, and make benchmarks based on those), but for now, I think this is the best LLM ranking that exists, because it's more of a test of how well the LLM can use tools, and it more closely aligns with my personal experience of how useful different LLMs are in day to day usage. Here's the ranking from the AgentBench paper: https://raw.githubusercontent.com/THUDM/AgentBench/main/assets/intro.png https://raw.githubusercontent.com/THUDM/AgentBench/main/asse...