To paraphrase a wise person, "All benchmarks are wrong, but some are useful". Benchmarking, like modelling or simulation, are one of many imperfect but useful

To paraphrase a wise person, "All benchmarks are wrong, but some are useful". Benchmarking, like modelling or simulation, are one of many imperfect but useful tools researchers can reach for when trying to better understand something. Benchmarks help us contextualize things. All benchmarks have biases, and some tell us more than others, but they certainly fall under a class of experimentation that falls under "research". Of course, it's important not to overstate findings. Performing well on any given coding benchmark does not tell us "this LLM is, broadly speaking, great at coding", but that it is great at coding as measured by a narrow testing set. There's also another snake pit to cross I'm that validation methodology is hard (data leakage, appropriate measures and stats, etc...) but plenty of quality research over comes that with rigour and patience.