Should these tests not be a black box? If you know these tests you can cheat on them right? So the idea of making a black box benchmark is that a closed system

Should these tests not be a black box? If you know these tests you can cheat on them right? So the idea of making a black box benchmark is that a closed system randomly chooses tests to run(without you knowing which ones or there nature) and then running them and just giving you the results as a composite metric to tell you how good the engine was. And put in an extra restriction that you can't test more than once every week(say), or some time frame like that. like that so that no one can test infinitely to game it. Folks into machine learning would know what i am talking about here right?