Honestly every time I see someone use MMLU as a benchmark I am wondering what they are trying to prove. MMLU is a simple multiple choice test with the answers b

Honestly every time I see someone use MMLU as a benchmark I am wondering what they are trying to prove. MMLU is a simple multiple choice test with the answers being available. Simply picking answers at random should give you 25 points. Knowing 50% of the answers and picking the rest randomly gives you 62.5%, which is very close to the scores of SOTA LLMs. The benchmarks that supposedly show reasoning are pretty bad and have very little to do with reasoning. A lot of the questions can be answered through memorization. I agree with you. The benchmarks are garbage. I thought about building my own benchmarks, but this would require building a complex benchmarking framework first and I just don't have the time for preparatory work like that.