Cross-validation (and the bootstrap, a related tool) is a useful technique. It's most useful when deriving the parameters for the model, though, rather than fo

Cross-validation (and the bootstrap, a related tool) is a useful technique. It's most useful when deriving the parameters for the model, though, rather than for confirming the validity of the model. Basically, if you have a strong prior reason to believe that the model is basically valid, but you need to know the parameters that fit the data. The more hypotheses that are tested using the 30% "test" dataset, the more that dataset inadvertently becomes a "training" dataset, by progressively invalidating models. For applications with real-world significance, you usually have to accept that the "test" dataset will expire after some number of experiments into the "training" dataset, and gather a new test dataset to prevent overfitting. Hastie, Tibshirani, and Friedman[1], which I heartily recommend, cover this extensively in chapter 7. [1] https://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf https://statweb.stanford.edu/~tibs/ElemStatLearn/printings/E...