2 ms·
People have been gaming ML benchmarks as long as there have been ML benchmarks. That's why it's better to see if other researchers are incorporating a techniqu
by hooloovoo_zoo 2y ago
People have been gaming ML benchmarks as long as there have been ML benchmarks. That's why it's better to see if other researchers are incorporating a technique into their actual models rather than 'is this paper the bold entry in a benchmark table'. But it takes longer.
- jandrese 2y agoWhen a measure becomes a target it is no longer a good measure. These ML benchmarks were never going to last very long. There is far too much pressure to game them, even unintentionally.