4 ms·
I like presenting the correlation of results between different benchmarks - I'd be interested in hearing to what extent this problem exists in more traditional
by phdelightful 5y ago
I like presenting the correlation of results between different benchmarks - I'd be interested in hearing to what extent this problem exists in more traditional benchmarking. One difference is that ML has this accuracy/quality component where in the past we've been more concerned with performance. Unfortunately this paper doesn't really address the long history of non-ML benchmarking, and I find it hard to believe no one has previously addressed the fragility of benchmark results.
- version_five 5y agoWhat are some non-ML benchmarks? Or put another way, what domain are you referring to? Some examples I can think of are bit error rate for communication and full width at half maximum of the point spread function for image resolution. BER is a much more objective benchmark (even if alone it might miss power or some other factor). FWHM has it's own set of problems, because unless your an astronomer, the thing you're imaging probably isnt a point source. So you get into more subjective resolution phantoms, e.g. in medical imaging, or comparisons in how some part of Lena looks under different treatments. But mostly,these benchmarks are all much simpler and objective than something like imagenet that has so much randomness in it, and no real underlying principle other than it being diverse enough to be general. I'm curious if there are other similarly random benchmarks used in other domains?