I've done research in speech recognition, a branch of machine learning, and analyzing the same data over and over again is the classic blunder. It is called "te

I've done research in speech recognition, a branch of machine learning, and analyzing the same data over and over again is the classic blunder. It is called "testing on the training data" and it over-estimates accuracy to an uncontrollable degree. People do try to avoid this by splitting their corpus into two, training on one half and testing on the other half, which works the first few times. Pretty soon though you have tuned your methods to get good results on the second half of the corpus and it is used up as a source of new data for an independent assessment of accuracy. There is a technique called "jack knifing" that sometimes helps a little. If the Vapnik–Chervonenkis dimension of your model is low enough then success on the training data does guarantee success on fresh test data. This never works in practice. Getting decent results always requires lots of tunable parameters and your VC dimension ends up way too high. Machine learning only works, to the extent that it works at all, because many of your parameters only affect part of your model. You have an effective VC dimension much lower than the parameter count indicates, so you do get some generalisation, but if you want to know you have to gather fresh test data. If you are doing research in machine learning you have to have a rolling program of data collection so that you can do genuine forecast testing. Hindcasting, that is to say "testing on the training data" is only really useful for finding bugs in your software. You should always be able to get good hindcast accuracy by dialing up the model dimension. If you cannot, you will never get good forecast accuracy. If you can get good hindcast accuracy that basically means that your software has passed one kind of bug test. You don't know your forecast accuracy until you gather fresh data. There is a human side to this. Doing research in machine learning is a ordeal of perpetual disappointment. Your research runs ahead of data collection. So you measure your hindcast accuracy. It is great. You have broken through. Then fresh data comes and you measure your forecast accuracy. Terrible disappointment. Science is hard!