I've always had this question, so I figured I'd ask it here: these large models are trained on enormous quantities of text - here, 780B tokens, which correspond

I've always had this question, so I figured I'd ask it here: these large models are trained on enormous quantities of text - here, 780B tokens, which corresponds to around 1-2 TiB of plaintext scraped from all over the web, using their tokenization scheme. How careful are researchers about making sure their test datasets don't end up in the model? How specifically do they ensure that this doesn't happen? EDIT: It looks like they specifically address in the section "Dataset Contamination" on page 35 of the paper - it appears the performance on the clean subsets of certain tasks, "clean" because that data was confirmed not to be in the training data, is similar to the performance on those tasks when all the questions, including those in the training data, are included. I'd need to have an ML research comment to know if this a valid way to control for the effects of memorization.