It depends on how often you go to the well. This is why things like Kaggle have limitations on how often you can submit entries for evaluation. This is why some

It depends on how often you go to the well. This is why things like Kaggle have limitations on how often you can submit entries for evaluation. This is why some studies go with a three-way split of train-calibration-test. And you can look at things like cross validation, boot-strapping etc. Nothing is perfect. As soon as you lock yourself into a one-size-fits-all way of validating your out-of-sample validity, you'll find lots of people who are optimizing for whatever test you come up with, not out-of-sample validity. It pays to pay attention to how well the researcher understands and deals with their particular set of problems, not how well they meet some arbitrary criteria. (Which is exactly how we ended up having all these problems with p-values! P-values aren't inherently bad, they're just applied indiscriminately because It's The Way We Do Things. If we start indiscriminately applying test/train splits we'll end up at a different place with its own set of problems. The chief problem is the lack of discrimination, not the specific tools under consideration.)