Academia is basically "self-reporting on the honor system". It works generally but there are lots of holes. Ultimately, "trust but verify" is necessary to avoid

Academia is basically "self-reporting on the honor system". It works generally but there are lots of holes. Ultimately, "trust but verify" is necessary to avoid getting caught in a wave of hype, or at least having someone in your own "circle of trust" say it works. This system leads naturally to elitism and a bunch of other problems which are seen in academia, but it seems better than the current alternatives to me given the current rabid focus on exact percentage score instead of quality/utility of an idea. The "right way" to do it is test once only per model/paper. If you are interested there are a huge number of sneaky ways overfitting can happen in ML [1]. Also interesting that you too see crypto and ML as related - I see them as opposites of the same coin. One tries to pull signal out of noise, the other tries to bury the signal in noise... but special noise. [1] http://hunch.net/?p=22 http://hunch.net/?p=22