Obviously if there has been a 'meta' process of refinement, such that the test set has been used in model development, then its not a clean test set any more, a

Obviously if there has been a 'meta' process of refinement, such that the test set has been used in model development, then its not a clean test set any more, and shouldn't be regarded as such. That's something for any researchers to watch out for, and be sure they don't do. And good researchers are well aware of these pitfalls. That's why I mentioned the validation sets, and that the test set must never have been looked at, or used before. But the point stands - if the method works on a clean test set, even if the test set is in the past, then it should be taken seriously. Would I trust such a model to predict the stock market in Q2 2010? No, because my prior belief is that the stock market is very hard to predict, so I would need very strong evidence to the contrary. But that has nothing to do with having confidence in models that have been tested on historical data.