8 ms·

Such aggressive usage of the test data set in determining the tuning parameters in effect makes your test data set part of your training data set. The more tim

• aelaguiz 14y ago
  Yep, I've definitely seen that. I bet if I were to reserve a portion of the dataset for validation after all tuning, I'd get a much better measure of it's actual ability at prediction. Using it in the way that I did, I wouldn't be surprised if there was significant overfitting. In fact I'd be surprised if there wasn't.
• gersh 14y ago
  You can split it into training, validation, and test sets if you have enough data. Then, the validation set would be used for the training. Presumably by using a genetic algorithm, he isn't going for a perfect fit, which may help avoid overfitting.
• brador 14y ago
  Question: Why is the data for machine learning split into a training dataset and a test dataset? Wouldn't using the entire dataset to build the model result in greater accuracy of the predictions?
  • aelaguiz 14y ago
    For this I actually had sufficient data available that adding more didn't do much. You need a testing dataset in order to validate the performance of the model. If you validate against the training set really what you're doing is measuring the model's ability to fit the training data - which it will be able to do with high accuracy. That will, however, result in a much diminished ability to predict any new games - as it isn't "learning" the features of college basketball as much as it is memorizing the contents of the training set. As others pointed out, it'd probably even be better to add a third grouping which is tested against only after the algorithm has finished - as an objective validation against as of yet unseen data.
  • numlocked 14y ago
    To prevent overfitting. A common technique is to use cross-validation, where you train some percentage of the data (90%) and test on the remaining (10%), and cycle through which 10% you use as test (each train/test split is called a "fold"). Once you've identified a model that resists overfitting, you can train on all of the data. edit: As noted in the other reply, having a truly blind validation set is still ideal.
  • reginaldo 14y ago
    When you develop a model, for instance, when implementing a classifier, you supposedly want to apply the developed model to other data, i.e., data you don't have available during development. In many situations, it doesn't make sense to test your model only when it's put to make or influence decisions in the real world (although you have to test in the real world too). You'll want to test the predictions of your model on data you already have the actual results for. To test your model you'll split your data into data you know and will let the model know about (training dataset), and data you know but the model can't know about (test dataset). That way you can use the data the model doesn't know about to make controlled experiments and compare models (and, if your data is really representative of the real world, your mofrl comparison and the performance of your chosen model will hold). The moral of the story is: if you don't split your data, you won't have any idea of how it performs in the real world, you'll only know how it performs with data it already knew about.
    • brador 14y ago
      I understand this is the way it is done, however, it all feels a bit hand wavy and not how my gut tells me to do it. I'd get the entire dataset, set columns as variables, assign weight to each of these variables and process each weight in 0.1 increments. (So the final number of passes is 11^n where n is number of variables). I'd have something at the end of the row to know what was predicted right (+1) and what was predicted wrong (-1), sum this column. Hit run until optimal weights for each variable are found. I'd use the entire dataset to do this. Is there any mathematics on defining what % of the dataset should be training vs. testing or is it left to the analyst (like with confidence intervals (95% hypothesis testing etc.))?
      • abecedarius 14y ago
        Solomonoff induction is a principled but uncomputable way to do without a test set. There might be a useful approximation to it for your problem.
      • jberryman 14y ago
        > Is there any mathematics on defining what % of the dataset should be training vs. testing or is it left to the analyst (like with confidence intervals (95% hypothesis testing etc.))? I think you're thinking about this wrong. You need to set aside some of your data for testing/cross-validation, otherwise you have no way of knowing even if you have enough data to train your model! i.e. you want to get to the point where additional training data isn't getting you a better model, and if you're not at that point you should probably collect more data.
  • btilly 14y ago
    Suppose that I have 2 possible models that I might try to build. I can train them both on my data set, but how do I get a sense as to which is better? When you have the split, you can then tell which idea seems to work better.
  • tel 14y ago
    Yes. Training on the test set overestimates performance, cross validation underestimates it.