3 ms·

I find newbs in ml don't appreciate cross validation. That's the one main trick. Keep some data out of the learning process to test an approaches ability on dat

I find newbs in ml don't appreciate cross validation. That's the one main trick. Keep some data out of the learning process to test an approaches ability on data it has not seen. With this one trick you can determine which algorithm is best, and the parameters. Advanced stuff like Bayes means you don't need it, but for your own sanity you should still always cross validate. Machine learning is about generalisation to unseen examples, cross validation is the metric to test this. Machine learning is cross validation.

jcutrell 13y ago
This was of utmost importance in my ERP class at GaTech - implementing a data segmenting algorithmic solution always requires that you hold out a sample set. This really is across all statistics, if I'm not mistaken - the validation set and the holdout.
syllogism 13y ago
First, you're always going to need to evaluate supervised models on some data you didn't train on. No method will ever relieve the need for that. Second, I never use cross-fold if I can help it. It's a last resort if the data's really small. Otherwise, use two held-out sets, one for development, and another for testing. Cross-fold actually really sucks! The small problem is that cross-validation has you repeating everything N times, when your experiment iteration cycle is going to be a big bottleneck. Second, it can introduce a lot of subtle bugs. Let's say you need to read over your data to build some sort of dictionary of possible labels for the instances, given one of the attributes. This can save you a lot of time. But if you're cross-validating, you can forget to do these pre-processes cross-validated too, and so perform a subtle type of cheating. Coding that stuff correctly often has you writing code into the run-time about the cross-fold validation. Nasty. With held-out data, you can ensure your run-time never has access to the labels. It writes out the predictions, and a separate evaluation script is run. This way, you know your system is "clean". Finally, when you're done developing, you can do a last run on a test set. Bonus example of cross-fold problems: Sometimes your method assumes independence between examples, even of this isn't strictly true. For instance, if you're doing part-of-speech tagging, you might assume that sentences in your documents are independent of each other (i.e. you don't have any cross-sentence features). But they're not _actually_ independent! If you cross-fold, and get to evaluate on sentences from the same documents you've trained on, you might do much better than in a realistic evaluation, where you have to evaluate on fresh documents.
- tlarkworthy 13y ago
  Yes I realise the importance of a testing set too. I see that as just another level validation. Hierarchies of left out data. If you understand in your heart what cross validation is, then the step to using a testing set is a trivial extension. I brought up cross validation in particular, because I see PhDs eating books on advanced techniques like structure learning. But they have no practical experience. And they ignore the basics for so long, as they don't realise validation is the most important bit. If you have not drawn the curve of training set error, validation error against training time, you have no done machine learning yet... This article doesn't mention it either :(

jcutrell 13y ago

This was of utmost importance in my ERP class at GaTech - implementing a data segmenting algorithmic solution always requires that you hold out a sample set. This really is across all statistics, if I'm not mistaken - the validation set and the holdout.

syllogism 13y ago

First, you're always going to need to evaluate supervised models on some data you didn't train on. No method will ever relieve the need for that. Second, I never use cross-fold if I can help it. It's a last resort if the data's really small. Otherwise, use two held-out sets, one for development, and another for testing. Cross-fold actually really sucks! The small problem is that cross-validation has you repeating everything N times, when your experiment iteration cycle is going to be a big bottleneck. Second, it can introduce a lot of subtle bugs. Let's say you need to read over your data to build some sort of dictionary of possible labels for the instances, given one of the attributes. This can save you a lot of time. But if you're cross-validating, you can forget to do these pre-processes cross-validated too, and so perform a subtle type of cheating. Coding that stuff correctly often has you writing code into the run-time about the cross-fold validation. Nasty. With held-out data, you can ensure your run-time never has access to the labels. It writes out the predictions, and a separate evaluation script is run. This way, you know your system is "clean". Finally, when you're done developing, you can do a last run on a test set. Bonus example of cross-fold problems: Sometimes your method assumes independence between examples, even of this isn't strictly true. For instance, if you're doing part-of-speech tagging, you might assume that sentences in your documents are independent of each other (i.e. you don't have any cross-sentence features). But they're not _actually_ independent! If you cross-fold, and get to evaluate on sentences from the same documents you've trained on, you might do much better than in a realistic evaluation, where you have to evaluate on fresh documents.

tlarkworthy 13y ago
Yes I realise the importance of a testing set too. I see that as just another level validation. Hierarchies of left out data. If you understand in your heart what cross validation is, then the step to using a testing set is a trivial extension. I brought up cross validation in particular, because I see PhDs eating books on advanced techniques like structure learning. But they have no practical experience. And they ignore the basics for so long, as they don't realise validation is the most important bit. If you have not drawn the curve of training set error, validation error against training time, you have no done machine learning yet... This article doesn't mention it either :(