3 ms·
20 years ago, one of my professors pointed out during a review that "benchmark is not research." Now I see literally everywhere benchmarks with very basic concl
by kmac_ 2y ago
20 years ago, one of my professors pointed out during a review that "benchmark is not research." Now I see literally everywhere benchmarks with very basic conclusions. I'm not criticizing this particular work, but such publications are low-effort unless they point to something new.
- buildbot 2y agoBenchmarks aren’t research, but research is 75% benchmarks of some kind, in my experience at least. Have to know where you are to know where to go, and once you know where to go and go there, well you have to know where you are!
- deleted 2y ago[deleted]
- Der_Einzige 2y agoYour professor was wrong (and his H index would be low today): https://neurips.cc/Conferences/2024/CallForDatasetsBenchmarks https://neurips.cc/Conferences/2024/CallForDatasetsBenchmark...
- spookie 2y agoH-index means nothing when there's always a name under authors that didn't do anything ;)
- MPSimmons 2y agoIt seems useful to have peer reviewed processes for comparing benchmarks that other research can reference, though.
- jszymborski 2y agoTo paraphrase a wise person, "All benchmarks are wrong, but some are useful". Benchmarking, like modelling or simulation, are one of many imperfect but useful tools researchers can reach for when trying to better understand something. Benchmarks help us contextualize things. All benchmarks have biases, and some tell us more than others, but they certainly fall under a class of experimentation that falls under "research". Of course, it's important not to overstate findings. Performing well on any given coding benchmark does not tell us "this LLM is, broadly speaking, great at coding", but that it is great at coding as measured by a narrow testing set. There's also another snake pit to cross I'm that validation methodology is hard (data leakage, appropriate measures and stats, etc...) but plenty of quality research over comes that with rigour and patience.
- gessha 2y agoCV ML context: A benchmark by itself is not really interesting but it is a crucial part of novel research since you have to compare your contribution to something else.