4 ms·
Too bad they don't cite the paper "The Benchmark Lottery" (M. Dehghani et al, 2021) (https://arxiv.org/abs/2107.07002 https://arxiv.org/abs/2107.07002) >
by carschno 5y ago
Too bad they don't cite the paper "The Benchmark Lottery" (M. Dehghani et al, 2021) (https://arxiv.org/abs/2107.07002 https://arxiv.org/abs/2107.07002)
> The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes the notion of "a benchmark lottery" that describes the overall fragility of the ML benchmarking process. The benchmark lottery postulates that many factors, other than fundamental algorithmic superiority, may lead to a method being perceived as superior. On multiple benchmark setups that are prevalent in the ML community, we show that the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks, highlighting the fragility of the current paradigms and potential fallacious interpretation derived from benchmarking ML methods. Given that every benchmark makes a statement about what it perceives to be important, we argue that this might lead to biased progress in the community. We discuss the implications of the observed phenomena and provide recommendations on mitigating them using multiple machine learning domains and communities as use cases, including natural language processing, computer vision, information retrieval, recommender systems, and reinforcement learning.
Edit: By "they", I was actually referring to the linked article. Strangely, even the paper the article is about does not cite "The Benchmark Lottery" at all.
- stayfrosty420 5y agoSeems like a distinct issue to me, though there are obvious parellels and similar outcomes.
- carschno 5y agoAgree, I was not saying it was redundant. But I would have expected a reference, as that paper was the first related item that came to my mind.
- koheripbal 5y agoBecause AIs are relatively narrowly focused, they suffer greatly from limited data sets. More often than not, an AI will simply memorize test data rather than "learning". This impacts the benchmarking process, so I think it's relevant.
- mellavora 5y agoHey, you're giving away my secret for passing AWS certification exams!
- sea-shunned 5y agoIt's also a shame that this paper in turn doesn't cite "Testing Heuristics: We Have It All Wrong" (J. N. Hooker, 1995) (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.4249&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71....), which discusses these issues in much the same way. It's good to see in "The Benchmark Lottery", however, that they look more into specific tasks and their algorithmic rankings and provide some sound recommendations. One thing that I'd add (somewhat selfishly as it relates to my PhD work), is the idea of generating datasets that are deliberately challenging for different algorithms. Scale this across a test suite of algorithms, and their relative strengths and weaknesses become clearer. The caveat here is that it requires having a set of measures that quantify different types of problem difficulty, which depending on the task/domain can range from well-defined to near-impossible.
- gopher_space 5y agoI've been looking at parsing paragraph structure and have started thinking about a conceptual mechanical turk/e e cummings line in the sand where it's just going to be easier to pay some kid with a cell phone to read words for you. The working implementations I've seen are heavily tied to domain and need to nail down language, which isn't really a thing. Quantification is fascinating, it seems to be something I take for granted until I actually want to make decisions. It's like I'm constantly trying to forget that analog and digital are two totally separate concepts. I wouldn't really recommend reading Castaneda to anyone but he describes people living comfortably with mutually exclusive ideas in their head walled off by context, and I'd like that sort of understanding.