4 ms·
So what if you split the dataset to say 70-30, tried to find hypotheses in the the 70% dataset, then validate them with the 30% dataset. This is the process I
by theossuary 10y ago
So what if you split the dataset to say 70-30, tried to find hypotheses in the the 70% dataset, then validate them with the 30% dataset. This is the process I use often with machine learning, would it also apply here?
- andrewla 10y agoCross-validation (and the bootstrap, a related tool) is a useful technique. It's most useful when deriving the parameters for the model, though, rather than for confirming the validity of the model. Basically, if you have a strong prior reason to believe that the model is basically valid, but you need to know the parameters that fit the data. The more hypotheses that are tested using the 30% "test" dataset, the more that dataset inadvertently becomes a "training" dataset, by progressively invalidating models. For applications with real-world significance, you usually have to accept that the "test" dataset will expire after some number of experiments into the "training" dataset, and gather a new test dataset to prevent overfitting. Hastie, Tibshirani, and Friedman[1], which I heartily recommend, cover this extensively in chapter 7. [1] https://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf https://statweb.stanford.edu/~tibs/ElemStatLearn/printings/E...
- theossuary 9y agoThat's really good to know, I'm still just beginning to learn about data analytics, and the link you provided is already proving to be an interesting read, so thanks!