3 ms·
Not testing the model on a validation set doesn't mean it doesn't have predictive power... but if you don't test it, there's no way you can say it does have pre
by araneae 17y ago
Not testing the model on a validation set doesn't mean it doesn't have predictive power... but if you don't test it, there's no way you can say it does have predictive power. And saying it predicts the data 87% of the time when you're using the training set is in fact fraudulent.
I can't believe that no one called him on this before, or that he got away with this! Anyone who has take a college statistics course knows that you have to have separate sets for training and for validation. There are even a ton of statistical methods to deal with this problem with a small data set, i.e. bootstrapping your training set. There's no excuse.
In fact, I remember reading some Feynman lecture in which he specifically mentions how bad this is... anyone got a citation for me?
- eru 17y ago> In fact, I remember reading some Feynman lecture in which he specifically mentions how bad this is... anyone got a citation for me? It's interesting that we still long for citations of some authority figure --- even when we all agree. (And I, too, would like to see what Feynman said.)
- araneae 17y agoIt's more of a longing to figure out what the heck I'm remembering. This happens to me all the time with people, too... someone will look really familiar but I can't place them at all, and it drives me crazy.
- tokenadult 17y agoI remember reading some Feynman lecture in which he specifically mentions how bad this is... anyone got a citation for me? Perhaps you are referring to this passage of "Cargo Cult Science": "There is also a more subtle problem. When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition." http://calteches.library.caltech.edu/51/02/CargoCult.pdf http://calteches.library.caltech.edu/51/02/CargoCult.pdf (Caltech lecture series is primary source) http://www.lhup.edu/~DSIMANEK/cargocul.htm http://www.lhup.edu/~DSIMANEK/cargocul.htm (There are several online reprints of this lecture text in HTML, more searchable than .PDF format.) See http://en.wikipedia.org/wiki/Cargo_cult_science http://en.wikipedia.org/wiki/Cargo_cult_science for background and both of those citations.