There are benchmarks that are decided beforehand and similar sentences are removed from even the first stage of training. This is useful for tracking model perf

There are benchmarks that are decided beforehand and similar sentences are removed from even the first stage of training. This is useful for tracking model performance and comparing different choices. e.g. see section 'Contamination of downstream tasks' of [1]. Every decent AI lab does this, else the benchmark result couldn't be trusted. OpenAI publishes results of ~20 benchmarks[2] and it is safe to assume they have made reasonable attempt to remove it from training set [1]: https://arxiv.org/pdf/2107.06499 https://arxiv.org/pdf/2107.06499 [2]: https://openai.com/index/hello-gpt-4o/ https://openai.com/index/hello-gpt-4o/