Agreed. Admittedly the paper that you're speaking about was on the extreme level of "bugged". The results were beyond stellar, attracting a great deal of intere

Agreed. Admittedly the paper that you're speaking about was on the extreme level of "bugged". The results were beyond stellar, attracting a great deal of interest, and researchers who are usually very reserved quickly pointed out methodological flaws and showed through previous / current experimentation how broken it was. A friend noted that their process was so flawed it was potentially _worse_ than if you'd just trained on the test set. Your broader point is spot on however. My general hope is that people are wary when their results are strong^, ensuring that you don't get a good result via "cheating", so the majority of bugs are likely to harm performance. If a result is also not reproducible (i.e. "cheating" bug) it won't be used and built on - but if a result is bugged but reproducible (i.e. bug where performance was lower than it should be) then we can still move the field forward even in spite of these issues. ^ When I achieved state of the art for a task - especially given it was a huge jump in accuracy for a relatively small model compared to the previous state of the art - I spent many days sitting there double checking I hadn't accidentally cheated ;)