> I understand this is the way it is done, however, it all feels a bit hand wavy and not how my gut tells me to do it. I know what you mean, I felt the same wa

> I understand this is the way it is done, however, it all feels a bit hand wavy and not how my gut tells me to do it. I know what you mean, I felt the same way when I first learned about training and test sets. However... > Is there any mathematics on defining what % of the dataset should be training vs. testing or is it left to the analyst (like with confidence intervals (95% hypothesis testing etc.))? Indeed there is - the proper name for the statistical technique is "cross-validation" and much work has been done on the matter. The most common way to do it is "k-fold cross validation", which, when k=2 simply means to split your data in half randomly and use one for training and one for test - basically what the author did. The generalized k-fold cross-validation means that you slice up your data into k subsets, pick one to use as a test set and use the rest as a training set. Then repeat the process ("folding") k times, picking a different test set every time and averaging the results. This, of course, takes longer, but it has a better gut feel because your algorithm is "learning" from all the data rather than just whatever slice you originally chose as your training set. Wikipedia lists a few more common ways to do it: http://en.wikipedia.org/wiki/Cross-validation_(statistics)#Common_types_of_cross-validation http://en.wikipedia.org/wiki/Cross-validation_(statistics)#C... However, since the author's process involves tuning parameters after testing the algorithm against the test set, I'm not really sure how cross-validation would work... Maybe standard k-fold with three sets instead of two - a training set, an intermediate test set, and a true test set?