4 ms·
My personal heuristic on reward curves is the opposite - if they're separated it probably means the baseline is weak rather than that the new method is revoluti
by Imnimo 5y ago
My personal heuristic on reward curves is the opposite - if they're separated it probably means the baseline is weak rather than that the new method is revolutionary. Close curves probably means the paper is comparing to a strong sota method. A paper that barely clears a weak baseline would be unlikely to be accepted, so an incremental improvement is more likely to indicate a rigorous evaluation. Then again, as the author says, heuristics are imperfect, so who knows.
- lumost 5y agoThere’s a big bias in industry towards weak baselines. Rigorous evaluation is both time consuming and unrewarding prone to conversations like “you mean we could have just used off the shelf method X?!”. A consistent theme I’ve observed is that scientists who use weak baselines and data sets which are easy enough to reach 97%+ accuracy are the most likely to be perceived as doing good science.