I feel, if you use a pre-trained model to do these things without knowing the set intersection of the test and that dataset, makes it very tough to know weather

I feel, if you use a pre-trained model to do these things without knowing the set intersection of the test and that dataset, makes it very tough to know weather inference is in the transitive closure of generated text the models were trained on or weather they really improved. There was another approach to grounding LLMs the other day from Normal Computing the other day: https://blog.normalcomputing.ai/posts/2023-09-12-supersizing-transformers/supersizing-transformers.html https://blog.normalcomputing.ai/posts/2023-09-12-supersizing... in which they use Mosaic but they also did not mentioned that this was actually done. Sentient or not, I feel there should be a standard on aggressively filtering out overlap on training and test datasets for approaches like this.