First, you're always going to need to evaluate supervised models on some data you didn't train on. No method will ever relieve the need for that. Second, I nev

First, you're always going to need to evaluate supervised models on some data you didn't train on. No method will ever relieve the need for that. Second, I never use cross-fold if I can help it. It's a last resort if the data's really small. Otherwise, use two held-out sets, one for development, and another for testing. Cross-fold actually really sucks! The small problem is that cross-validation has you repeating everything N times, when your experiment iteration cycle is going to be a big bottleneck. Second, it can introduce a lot of subtle bugs. Let's say you need to read over your data to build some sort of dictionary of possible labels for the instances, given one of the attributes. This can save you a lot of time. But if you're cross-validating, you can forget to do these pre-processes cross-validated too, and so perform a subtle type of cheating. Coding that stuff correctly often has you writing code into the run-time about the cross-fold validation. Nasty. With held-out data, you can ensure your run-time never has access to the labels. It writes out the predictions, and a separate evaluation script is run. This way, you know your system is "clean". Finally, when you're done developing, you can do a last run on a test set. Bonus example of cross-fold problems: Sometimes your method assumes independence between examples, even of this isn't strictly true. For instance, if you're doing part-of-speech tagging, you might assume that sentences in your documents are independent of each other (i.e. you don't have any cross-sentence features). But they're not _actually_ independent! If you cross-fold, and get to evaluate on sentences from the same documents you've trained on, you might do much better than in a realistic evaluation, where you have to evaluate on fresh documents.