It is not people are performing validation on the test set, that would be straightly cheating. What #3 has described is a totally normal workflow: 1. Conceptu

It is not people are performing validation on the test set, that would be straightly cheating. What #3 has described is a totally normal workflow: 1. Conceptualize a new model idea 2. Implementing and training is totally legal without involving test set. 3. However, once finished, the model is evaluated on test set, the performance of which will decide whether this idea is worthy of publishing or not. If not, they go to the first step and repeat. Such loop essentially makes the test, the actual validation set, if you think human as his/her own optimizer, and he/she takes a look at the test set periodically, and decide whether to pursue the current idea or not. Sounds like early stopping, isn't it? Remember back in 2015, there is a debacle from Baidu, where a researcher had fabricated multiple accounts to run unlimited tests against ImageNet's own reserved test set, which the competition straightly forbade. If a test set is a 'true' test set, then it should work like a test in a real world: be kept secret before revealing to the public, once evaluated, the same problems/examples shall never appear in the later tests ever. But such approach would not be accepted because the cost is simply too high.