4 ms·

No I don't use the term overfitting for a model where the accuracy is getting better. I think it's misleading.

by jph00 3y ago

mjburgess 3y ago
Accuracy is very rarely a useful metric. It's more an engineering metric than something a user would ever care about. What users want is to have their own credences properly calibrated by engaging with some system. From a physics textbook, they want a systematic presentation of ideas which allows them to build intuitions etc. It's important to formulate the actual goal of the system, rather than just the engineer's goal (consider eg., "width of pipes" vs., "clean running water"). In the case of statistical AI systems, the goal is often best formulated in terms of the confidences of the system not its output. Since its output accuracy is kinda nonlinear and discontinuous in those confidences. So from a statical AI Q&A system we dont want The Answer, we want the system to have expert-like confidences over possible answers. Of course, as soon as you start formulating these metrics, all the SoA 99%+ accuracy hype evaporates. Since most of these systems have terrible confidence distributions. Consider, eg., ChatGPT whose answers are often plausibly accurate (they count as an answer) but just repeat some silicon valley hype in a way an expert wouldnt. ChatGPT rarely has the careful scepticism of an expert, rarely presents ideas in an even handed way, rarely mentions the opposite. It makes generating reference materials on areas with expert disagreement quite dangerous. ChatGPT presents the non-expert credence distribution. (And indeed, always does, since it just models (Q,A) frequencies which are not truth-apt)
- DougBTX 3y ago
  This is mixing two meanings of confidence which could lead to confusion. The OP is using confidence to describe how high the per-token probability scores are, while you are talking about the confidence expressed in the tone of voice of the language generated by the model. Really those are orthogonal issues. (Eg, a model could predict with high probability that a output should be “I don’t know”)
  - mjburgess 3y ago
    It seems like i'm mixing them, but i'm not. I'm saying as a matter of fact ChatGPT should have different confidences in propositions. My issue isnt the tone of voice, my issue is the content of what it's saying is wrong wrt what we care about, ie., expert credences (/confidences) in the claims it's generating. It can "express confidently" scepticism; it does not. That's the issue. In my lang above i was mostly using credence to talk about the strength of the mental state of belief; and confidence to talk about the model of that used in statistical AI.
Nevermark 3y ago
Over fitting literally means what it says, fitting the training data too well to maintain a well formed function. This is many decades old terminology for a well established effect that occurs for all curve fitting, function approximation, parameter optimizing, and model training algorithms. You can Google it with no other context: “over fitting”. [0] “Confidence” isn’t its name and its meaning has nothing to do with the effect. Nothing wrong with making up terminology for new effects, but this one is an oldie. [0] https://www.google.com/search?q=over+fitting&ie=UTF-8&oe=UTF-8&hl=en-us&client=safari https://www.google.com/search?q=over+fitting&ie=UTF-8&oe=UTF...
- Nevermark 3y ago
  Addendum: Most data has a fraction which is noise, or is dependent on information not completely captured by training data. In both cases, training to a perfect fit of training data makes no sense. Learning the particular noise in training data guarantees worse results on real data, where the noise will, by definition, be different. The same is true for exactly reproducing data that depended on some unaccounted for information. Once applied, the unaccounted information in the training data will have no predictive quality. So perfectly fitting data is usually a terrible idea, even when it can be done. Training data captures a problem to be solved, as best it can. But it isn’t the same as the actual problem. —— Best practice: 1. Use training data to optimize a model 2. Use separate validation data to approximate current generalization quality, and stop at (or revert to) the point where validation performance was best. 3. Use separate test data, not used for model design in any way, for a completely independent approximation of generalization performance. Wide disparities between validation & test performance suggest problems. Ideally they should track each other fairly closely. If not, more data is probably needed to characterize the problem more reliably. NEVER just retrain, or tweak training parameters, until training & the validation stop produce good test performance. That means the test data was actually used in the design and is no longer an independent measure of performance! —— Assuming you are having trouble getting similar validation & test performances: A good way to ensure test performance is real, is to retrain the model in exactly the same way, several times, with different random divisions of training, validation & test data. If test results are good regardless of data divisions, then they are reliable. Then you can eliminate the test performance dependency, by randomly selecting any of the models. Don’t choose the one with best test results! That takes discipline! In that case, the mean of all test performances is your best estimate of generalization performance, regardless of model. (Throwing out models with the worst and best test performances is ok, it avoids outlier training failures or successes equally. Note I said “avoids”, not “eliminates”, since the worst & best test performances are estimates of generalization, not a measure of actual generalization.)

No I don't use the term overfitting for a model where the accuracy is getting better. I think it's misleading.

by jph00 3y ago

No I don't use the term overfitting for a model where the accuracy is getting better. I think it's misleading.

Accuracy is very rarely a useful metric. It's more an engineering metric than something a user would ever care about. What users want is to have their own credences properly calibrated by engaging with some system. From a physics textbook, they want a systematic presentation of ideas which allows them to build intuitions etc. It's important to formulate the actual goal of the system, rather than just the engineer's goal (consider eg., "width of pipes" vs., "clean running water"). In the case of statistical AI systems, the goal is often best formulated in terms of the confidences of the system not its output. Since its output accuracy is kinda nonlinear and discontinuous in those confidences. So from a statical AI Q&A system we dont want The Answer, we want the system to have expert-like confidences over possible answers. Of course, as soon as you start formulating these metrics, all the SoA 99%+ accuracy hype evaporates. Since most of these systems have terrible confidence distributions. Consider, eg., ChatGPT whose answers are often plausibly accurate (they count as an answer) but just repeat some silicon valley hype in a way an expert wouldnt. ChatGPT rarely has the careful scepticism of an expert, rarely presents ideas in an even handed way, rarely mentions the opposite. It makes generating reference materials on areas with expert disagreement quite dangerous. ChatGPT presents the non-expert credence distribution. (And indeed, always does, since it just models (Q,A) frequencies which are not truth-apt)

DougBTX 3y ago
This is mixing two meanings of confidence which could lead to confusion. The OP is using confidence to describe how high the per-token probability scores are, while you are talking about the confidence expressed in the tone of voice of the language generated by the model. Really those are orthogonal issues. (Eg, a model could predict with high probability that a output should be “I don’t know”)
- mjburgess 3y ago
  It seems like i'm mixing them, but i'm not. I'm saying as a matter of fact ChatGPT should have different confidences in propositions. My issue isnt the tone of voice, my issue is the content of what it's saying is wrong wrt what we care about, ie., expert credences (/confidences) in the claims it's generating. It can "express confidently" scepticism; it does not. That's the issue. In my lang above i was mostly using credence to talk about the strength of the mental state of belief; and confidence to talk about the model of that used in statistical AI.

Nevermark 3y ago

Over fitting literally means what it says, fitting the training data too well to maintain a well formed function. This is many decades old terminology for a well established effect that occurs for all curve fitting, function approximation, parameter optimizing, and model training algorithms. You can Google it with no other context: “over fitting”. [0] “Confidence” isn’t its name and its meaning has nothing to do with the effect. Nothing wrong with making up terminology for new effects, but this one is an oldie. [0] https://www.google.com/search?q=over+fitting&ie=UTF-8&oe=UTF-8&hl=en-us&client=safari https://www.google.com/search?q=over+fitting&ie=UTF-8&oe=UTF...

Nevermark 3y ago
Addendum: Most data has a fraction which is noise, or is dependent on information not completely captured by training data. In both cases, training to a perfect fit of training data makes no sense. Learning the particular noise in training data guarantees worse results on real data, where the noise will, by definition, be different. The same is true for exactly reproducing data that depended on some unaccounted for information. Once applied, the unaccounted information in the training data will have no predictive quality. So perfectly fitting data is usually a terrible idea, even when it can be done. Training data captures a problem to be solved, as best it can. But it isn’t the same as the actual problem. —— Best practice: 1. Use training data to optimize a model 2. Use separate validation data to approximate current generalization quality, and stop at (or revert to) the point where validation performance was best. 3. Use separate test data, not used for model design in any way, for a completely independent approximation of generalization performance. Wide disparities between validation & test performance suggest problems. Ideally they should track each other fairly closely. If not, more data is probably needed to characterize the problem more reliably. NEVER just retrain, or tweak training parameters, until training & the validation stop produce good test performance. That means the test data was actually used in the design and is no longer an independent measure of performance! —— Assuming you are having trouble getting similar validation & test performances: A good way to ensure test performance is real, is to retrain the model in exactly the same way, several times, with different random divisions of training, validation & test data. If test results are good regardless of data divisions, then they are reliable. Then you can eliminate the test performance dependency, by randomly selecting any of the models. Don’t choose the one with best test results! That takes discipline! In that case, the mean of all test performances is your best estimate of generalization performance, regardless of model. (Throwing out models with the worst and best test performances is ok, it avoids outlier training failures or successes equally. Note I said “avoids”, not “eliminates”, since the worst & best test performances are estimates of generalization, not a measure of actual generalization.)