4 ms·

Over fitting literally means what it says, fitting the training data too well to maintain a well formed function. This is many decades old terminology for a we

Over fitting literally means what it says, fitting the training data too well to maintain a well formed function. This is many decades old terminology for a well established effect that occurs for all curve fitting, function approximation, parameter optimizing, and model training algorithms. You can Google it with no other context: “over fitting”. [0] “Confidence” isn’t its name and its meaning has nothing to do with the effect. Nothing wrong with making up terminology for new effects, but this one is an oldie. [0] https://www.google.com/search?q=over+fitting&ie=UTF-8&oe=UTF-8&hl=en-us&client=safari https://www.google.com/search?q=over+fitting&ie=UTF-8&oe=UTF...

Nevermark 3y ago
Addendum: Most data has a fraction which is noise, or is dependent on information not completely captured by training data. In both cases, training to a perfect fit of training data makes no sense. Learning the particular noise in training data guarantees worse results on real data, where the noise will, by definition, be different. The same is true for exactly reproducing data that depended on some unaccounted for information. Once applied, the unaccounted information in the training data will have no predictive quality. So perfectly fitting data is usually a terrible idea, even when it can be done. Training data captures a problem to be solved, as best it can. But it isn’t the same as the actual problem. —— Best practice: 1. Use training data to optimize a model 2. Use separate validation data to approximate current generalization quality, and stop at (or revert to) the point where validation performance was best. 3. Use separate test data, not used for model design in any way, for a completely independent approximation of generalization performance. Wide disparities between validation & test performance suggest problems. Ideally they should track each other fairly closely. If not, more data is probably needed to characterize the problem more reliably. NEVER just retrain, or tweak training parameters, until training & the validation stop produce good test performance. That means the test data was actually used in the design and is no longer an independent measure of performance! —— Assuming you are having trouble getting similar validation & test performances: A good way to ensure test performance is real, is to retrain the model in exactly the same way, several times, with different random divisions of training, validation & test data. If test results are good regardless of data divisions, then they are reliable. Then you can eliminate the test performance dependency, by randomly selecting any of the models. Don’t choose the one with best test results! That takes discipline! In that case, the mean of all test performances is your best estimate of generalization performance, regardless of model. (Throwing out models with the worst and best test performances is ok, it avoids outlier training failures or successes equally. Note I said “avoids”, not “eliminates”, since the worst & best test performances are estimates of generalization, not a measure of actual generalization.)

Nevermark 3y ago

Addendum: Most data has a fraction which is noise, or is dependent on information not completely captured by training data. In both cases, training to a perfect fit of training data makes no sense. Learning the particular noise in training data guarantees worse results on real data, where the noise will, by definition, be different. The same is true for exactly reproducing data that depended on some unaccounted for information. Once applied, the unaccounted information in the training data will have no predictive quality. So perfectly fitting data is usually a terrible idea, even when it can be done. Training data captures a problem to be solved, as best it can. But it isn’t the same as the actual problem. —— Best practice: 1. Use training data to optimize a model 2. Use separate validation data to approximate current generalization quality, and stop at (or revert to) the point where validation performance was best. 3. Use separate test data, not used for model design in any way, for a completely independent approximation of generalization performance. Wide disparities between validation & test performance suggest problems. Ideally they should track each other fairly closely. If not, more data is probably needed to characterize the problem more reliably. NEVER just retrain, or tweak training parameters, until training & the validation stop produce good test performance. That means the test data was actually used in the design and is no longer an independent measure of performance! —— Assuming you are having trouble getting similar validation & test performances: A good way to ensure test performance is real, is to retrain the model in exactly the same way, several times, with different random divisions of training, validation & test data. If test results are good regardless of data divisions, then they are reliable. Then you can eliminate the test performance dependency, by randomly selecting any of the models. Don’t choose the one with best test results! That takes discipline! In that case, the mean of all test performances is your best estimate of generalization performance, regardless of model. (Throwing out models with the worst and best test performances is ok, it avoids outlier training failures or successes equally. Note I said “avoids”, not “eliminates”, since the worst & best test performances are estimates of generalization, not a measure of actual generalization.)