for me the key lies in the experiment. only by generating new data through new experiments, do we receive robust theories. otherwise we overfit to whatever data

for me the key lies in the experiment. only by generating new data through new experiments, do we receive robust theories. otherwise we overfit to whatever data is at hand. hence why we need cross validation when working with canned data.