Funny, that this idea is so foreign to ML. As a macromolecular crystallographer, Rfree [0] is something drilled into every student's brain from day one! TL;DR:

Funny, that this idea is so foreign to ML. As a macromolecular crystallographer, Rfree [0] is something drilled into every student's brain from day one! TL;DR: Randomly, a certain percent (5-10%) of data is 'hidden' and never used for building/refining your model, but is only used to evaluate how well your model fits (or explains) that unseen data. This is absolutely, fundamentally essential to prevent over-fitting your data!! EDIT: Think that you are solving a huge jigsaw puzzle, but made of thousands of jello pieces. You randomly hide a 100 or so pieces and try to solve the puzzle. Having used all the pieces (except the hidden 100), you think the puzzle forms a Treasure Map. Now, you take the previously hidden pieces and try to fit those into the puzzle and if after using the hidden pieces your puzzle still looks like a Treasure Map, you may have found a (mostly) correct solution. But, if you are unable to fit those hidden places in a way that still keeps the Treasure Map intact, you must question if you did in fact find the correct solution or if there is another, slightly different, solution that may be (more) correct because it will account for the hidden pieces a little better? [0] http://reference.iucr.org/dictionary/Free_R_factor http://reference.iucr.org/dictionary/Free_R_factor