3 ms·

When is a model ever evaluated based on its ability to predict the training data? That is classic overfitting. This is why the training set is kept separate f

When is a model ever evaluated based on its ability to predict the training data? That is classic overfitting. This is why the training set is kept separate from test/validation sets. https://en.wikipedia.org/wiki/Training,_test,_and_validation_sets https://en.wikipedia.org/wiki/Training,_test,_and_validation...

nl 9y ago
I think you missed the point here. The author is showing exactly why this is a big problem in NLP, beyond the test/train split issue. Even with test/train splits you still need to smooth your model's predictions because it will always get unseen words. That's just the nature of word based lanugage models.
rspeer 9y ago
Everyone capable of understanding this thread knows what overfitting is, and the difference between training and test data. That is the topic we are talking about. Because test data is unseen at training time, it cannot possibly affect the way your model is trained. The test data is not how you prevent overfitting, it's how you measure overfitting. To prevent overfitting, you need to design features such as smoothing into your model.
- haberman 9y ago
  I think your point was a straw man. You can tell if your model is good by measuring its predictive power on data outside your training set. You seemed to be arguing that only way to know the model is wrong is to insert a human who just knows a priori that the model is wrong and what "correction" to apply. But I think this is a false analysis. You can measure a model's accuracy by seeing how well it predicts data outside the training set. If the model is highly predictive, you have a good model. It doesn't take a human's subjective analysis or principled intervention to determine whether the model is accurate with respect to the available data. This article concerns cases where the model is admittedly accurate (with respect to the available data), but is subjectively considered objectionable. That is a totally different issue.

nl 9y ago

I think you missed the point here. The author is showing exactly why this is a big problem in NLP, beyond the test/train split issue. Even with test/train splits you still need to smooth your model's predictions because it will always get unseen words. That's just the nature of word based lanugage models.

rspeer 9y ago

Everyone capable of understanding this thread knows what overfitting is, and the difference between training and test data. That is the topic we are talking about. Because test data is unseen at training time, it cannot possibly affect the way your model is trained. The test data is not how you prevent overfitting, it's how you measure overfitting. To prevent overfitting, you need to design features such as smoothing into your model.

haberman 9y ago
I think your point was a straw man. You can tell if your model is good by measuring its predictive power on data outside your training set. You seemed to be arguing that only way to know the model is wrong is to insert a human who just knows a priori that the model is wrong and what "correction" to apply. But I think this is a false analysis. You can measure a model's accuracy by seeing how well it predicts data outside the training set. If the model is highly predictive, you have a good model. It doesn't take a human's subjective analysis or principled intervention to determine whether the model is accurate with respect to the available data. This article concerns cases where the model is admittedly accurate (with respect to the available data), but is subjectively considered objectionable. That is a totally different issue.