>"What the paper is saying, is as a whole their basket of terms performed better than random strategies." This is a totally meaningless metric unless they took

>"What the paper is saying, is as a whole their basket of terms performed better than random strategies." This is a totally meaningless metric unless they took measures to blind themselves to the validation data. Did they do that, or did they try out a bunch of different things, do interim analysis on the performance, etc (as is almost always done in academia)? If the latter, all this "test" amounts to saying you checked if A=3, but consciously set A=5. Disproving such a hypothesis has zero value... https://www.kaggle.com/wiki/Leakage https://www.kaggle.com/wiki/Leakage