I applaud the author of this post. I've seen a lot of people suffer with machine learning because they don't understand this basic concept. Taking MOOC classes

I applaud the author of this post. I've seen a lot of people suffer with machine learning because they don't understand this basic concept. Taking MOOC classes and reading textbooks is a great way to learn, but they tend to focus on a lot on the mathematical principle and not the start-from-nothing practical considerations. Machine learning is almost like learning chess in that there are certain obvious mistakes that noobs continue to make. And like chess there are multiple levels of thinking and understanding that are almost impossible to teach to someone that doesn't have lots of experience. Hopefully more blog posts like this will help people get past the novice level. Regarding technical content: N-fold Cross validation [1] can be a more effective approach to having a single held out or validation set. You split your data into N groups, say N = 10. Then you use groups 2-10 as a training set to make predictions on group 1, then groups 1,3-10 to make predictions on group 2, etc. Recombine the prediction output files and use the measured error to tune and tweak your predictor. It's more work and can still lead to overfitting, but it's generally better to overfit the entire training set than it is to overfit one held out sample. [1] http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29 http://en.wikipedia.org/wiki/Cross-validation_%28statistics%...